In middleware integration we generally face a issue how to handle large data of flatfiles and xml in the case of large files(suppose greater than 20MB) , it will slowdown the Integration server if directly loaded into the server , if the file is of huge size (greater than 100MB) then in this case it might crash the server . So, in order to save ourself from this situation we must take a precautive measure , this can be done by streaming the file into the server . So, in this blog we will see basically how to do large file handling in Integration server in the case of both Flat File and xml files.
Large File Handling (Flat Files)
Now we will see how to do large file handling in the case of flat files.
we can use webMethods FilePolling Port , pub.file:getFile , or using FTP (i.e using this service pub.client:ftp) to get the file depending on whether the file is on localsystem or remote FTP system w.r.t to webMethods Integration server(IS).
webMethods FilePolling port by default streams the file into the IS , for pub.file:getFile select the optionsal input parameter as load as stream and it will stream the file into the server .
Normally we use pub.flatFile:convertToValues service to convert the flatfile into IS document , this time also we will use the same service with slight modification we will set the iterate input parameter as true.
pub.flatFile:convertToValues: This service is used in convert the flat file data to IS Document. Iterate = true value ensures that the records from the flat file is read one By one. Output ffIterator is passed at the input of the next invocation of this service.
If ffIterator is null, exit from the loop. It means all the available records in the flat file are processed.
pub.xml:xmlNodeToDocument : This service converts the XMLNode to Document
Now lets see the structure of the main flow service where Large file Handling for xml has been done .
Large File Handling (Flat Files)
Now we will see how to do large file handling in the case of flat files.
we can use webMethods FilePolling Port , pub.file:getFile , or using FTP (i.e using this service pub.client:ftp) to get the file depending on whether the file is on localsystem or remote FTP system w.r.t to webMethods Integration server(IS).
webMethods FilePolling port by default streams the file into the IS , for pub.file:getFile select the optionsal input parameter as load as stream and it will stream the file into the server .
Normally we use pub.flatFile:convertToValues service to convert the flatfile into IS document , this time also we will use the same service with slight modification we will set the iterate input parameter as true.
pub.flatFile:convertToValues: This service is used in convert the flat file data to IS Document. Iterate = true value ensures that the records from the flat file is read one By one. Output ffIterator is passed at the input of the next invocation of this service.
If ffIterator is null, exit from the loop. It means all the available records in the flat file are processed.
Large File Handling XML
Generally in real time scenarion for large xml it has been seen that some nodes are there that are being repeated several tomes making the xml file large enough . The large file handling for xml in webMethods targets this point and in built services are present to hanle these kind of scenarios.
Let's first see the inbuilt services present to do large file handling in the case of xml.
pub.xml:xmlStringToXMLNode – This service converts the input from String to XML node , this is a general service used in webMethods where we want to convert xml string data to xml node . This service is good in built service having great performance it plays a vital role in large file handling.
pub.xml:getXMLNodeIterator : This service will get the node one by one according to the Criteria set in the pipeline. Over here the criteria is set as EmployeeDetails and OFCDetails. This is the main service used for large file handling for XML , Just like flat file service , this service also iterates over the node and process it one by one .
pub.xml:getNextXMLNode : This service is used in order to get the next node .
Now lets see the structure of the main flow service where Large file Handling for xml has been done .
If you all have any queries just comment here . This is all for Flat File Handling in webMethods IS.
Enjoy Reading , If you have any Question related to webMethods please comment below , I will try to answer the questions.Please Subscribe for latest update on the blogs
Enjoy Reading , If you have any Question related to webMethods please comment below , I will try to answer the questions.Please Subscribe for latest update on the blogs
30 Comments
Good Article .. It helped a lot
ReplyDeleteHi Rakesh Tiwari,
ReplyDeleteI have example about flat file as flow:
- step 1: read file using flat file
- step 2: insert info into DB.
I wrote a flow service:
-INVOKE pub.file:getFile
-INVOKE pub.flatFile:convertToValues
-INVOKE flatfile:insertCustomer (a flow serive using to insert to DB)
But when I run flow service, It insert only the first record. Can you help me resolve this problem? (I'm beginer in webmethods)
thanks you soo much!
If you have multiple records (i.e more than one row in your file) . Then you should do loop over the document and inside loop call the insertCustomer service. But this is not a good approach but this approach will solve your problem.
ReplyDeleteGood Help.. Rakesh.. :) u did agreat job..
ReplyDeleteHello Rakesh,
ReplyDeleteI need write the file in FTP.
Please suggest how I can achieve it.
Hello Asker ,
DeleteBy "I need to write the file in FTP" you mean that you need to write the file in a remote FTP location . If this is the case then you can use the pub.client:ftp services present in the WmPublic package.
what is value we will provide in ffiterator
ReplyDeleteHi ,
DeleteWhen you call the service pub.flatfile:convertToValues , then if you are setting the input iterate as "true" , then in the service out you will get the ffiterator object , pass the same object in the input of the pub.flatfile:convertToValues service.
Hi Rakesh,
DeleteYou mean to say we have to call pub.flatfile:convertToValues two times??
What would be the value of Count in repeat step if we will use iterate as a True?
Thank you.
No need to call the pub.flatfile:convertToValues twice , you have to call it only once inside the repeat step , just what you need to do is pass the ffiterator in the input of the pub.flatfile:convertToValues service during design time for better code readability and maintenance , anyways if you donot pass the ffiterator in input it will be implicitly mapped during runtime in the next iteration (unless you have not dropped it from the pipeline out).
DeleteNo need to set any value for the Count parameter in repeat step , just leave it as it is , only thing you have to set is the repeat on parameter set it as "Success".To exit from the repeat statement you have to add one branch step and set the switch variable as "ffiterator" and evaluate label as "false" and then add one exit step (exit from loop) when ffiterator is null (Please refer the above article I have mentioned with screenshot).This repeat statement will act as "while" loop in normal programming concepts.
Hi Rakesh,
ReplyDeleteThanks for your article, This is very informative.
I have a large file handling webMethods scenario, Please help with that.
Scenario:
Flat File Stucture:
B
value1
value2
value3
C (we are getting multiple records for C)
value_i
value_j
value_k
D
value_m
value_n
value_o
Here in B we have few values and C, D (C is repeating)
Can you please sugget that, how to handle this kind of flat file, i have above suggested method but it is not working.
Thanks,
Abdul
Hello Rakesh,
ReplyDeleteThis is a very helpful article. However, I didn't manage to make my service to work properly...
Once a week system has to process a really huge file (500 MBs) and even with this iteration property system "freezes".. I am making some tests to see what is the maximum file size the system is able to process... Did you make any test like that? Do you know if there is a way to specify a maximum number of lines to process by iteration?
Many thanks in advance,
If you have such a large file , then it's better to go with the "Divide and Conquer" approach. Divide the file into smaller files based on the record count. Read the smaller files one by one and publish the content and then in the subscribing service do whatever transformation you want to do.
DeleteThe advantages of this approach are :-
1. You will reading smaller files , thus the load on the server will be reduced significantly.
2. Decoupled approach , as you are publishing the contents and doing the processing of the file in the subscribing service.
3, You can spawn multiple threads in the subscribing service by keeping the trigger property to "concurrent" and max execution thread property to some optimum number , which you can decide by doing proper load testing.
how will you divide it into chunks?
DeleteHello Asker ,
DeleteIt depends on what type of file you have , is it possible to divide the file into individual files and then process each individual files independently.
For Flat files , we can divide a bigger file into smaller file based on the record count say include 10,0000 record per file.
Hello Rakesh,
ReplyDeleteCould you provide any documentation or screen shot to implement EDI large file handling in webMethods.
Hi Rakesh,
ReplyDeleteUpto what file size can IS handle?
What is the maximum file size IS can process without any restrictions?
Hello Asker ,
ReplyDeleteWhat file size can IS handle ? : The answer to this question is based on how much memory is allocated to the server.
What is the maximum file size IS can process without any restrictions? : It is always better to follow proactive approach to the file processing , as it is memory intensive operation.
In one of my project : If the file size was more than 20 MB , we used to treat file as large file and were streaming the content and using ffiterator concept of flat file.
Hi Rakesh,
ReplyDeleteDo you have any restrictions on the number of records to be read when reading it from DB via SQL. What would be the select query result set size- 5000, 10,000, 15,000?
How many records can you bring in your IS memory without any server slowdowns. Even if the JVM heap size is 8GB, we cannot read 100 MB records into IS at a time right?
I have EDI file with file size of 20mb and it's taking more that 4+ hours to completely.Could you please let me know how can we process it with lesser time
ReplyDeleteNice article. Thanks for sharing this.
ReplyDeleteHi All,
ReplyDeleteI tried creating a flow service for large flat file hanlding as suggested in the above first 2 snaps.
I was able to use the repeat,iterate and ffiterator as given above.
the values coming in the flat file DT one by one.
But when i am mapping the values to some other doc say, Doc1 , the fields are coming null.
I tried to insert the values to some DB by mapping the fields to Flat File DT to insert adapter. There also i got null values.
Can you please help me understand what could be the reason of it?
Thanks a lot bro : great tutorial so far i have seen.
ReplyDelete
ReplyDeleteI was scrolling the internet like every day, there I found this article which is related to my interest. The way you covered the knowledge about the subject and the 2 BHK flat in hoshangabad road was worth to read, it undoubtedly cleared my vision and thoughts towards B 4 BHK flat in hoshangabad road. Your writing skills and the way you portrayed the examples are very impressive. The knowledge about 3 BHK flat in hoshangabad road is well covered. Thank you for putting this highly informative article on the internet which is clearing the vision about top builders in Bhopal and who are making an impact in the real estate sector by building such amazing townships.
What are the repeat step properties?
ReplyDeleteHi Avinah Rai ,
DeleteI would recommend you to please go through the Service Development Help official documentation from Software AG
Hi Rakesh,
ReplyDeleteCould you please help me out with is casestudy.
Here is your case study assignment : Read a large file from a directory ( just for simulation use 500 KB size ) and split that file into 3 files of size ( 100 KB, 250 KB and 150 KB) and write into 3 different directories. Broker is optional.
did you get the solution? I am looking for this as well.
DeleteI am looking for this solution as well. Please let me know if you get this. Thanks.
DeleteI know i am too late to this.. please use the split command in the executeOSCommand service of webMethods for more details on the usage of split refer https://www.geeksforgeeks.org/split-command-in-linux-with-examples/ and also refer the built in services of webmethods document
Delete