Flat File Handling in webMethods

Flat File Handling in webMethods , this will be the topic of this blog . So, before we proceed we should have basic idea about flat files what they , how they are organised and how they are parsed .

So, what is a flatfile , A simple text file having data in text format(non formatted text) is called flatfile , flatfiles are used since the advent of computers in business and still they are used widely .

Flatfiles use the same concept used in excel and database i.e record, fields and composites .
Record :- A complete row of fields and composite is called a record.
Field:- We can call it a column describing the data.
Composite:- If a field is composed of many subfield , then that field is called composite , example can be address it can be made up of subfield like street,house number,pincode and etc.

How many different types of flatfiles are there?

Flatfiles can be broadly classified into three different types based on the type of their record parsing methods , these three types are :-

1)Delimited FlatFile
2)Fixed length FlatFile
3)Variable length FlatFile

Again Flatfiles can be divided into two different types based on the type of field extractors , these are :-
1)Fixed length
2)Nth field

In Fixed length extractor type each field is defined by its starting index and end index , in this case the length of the field will be consatnt for every record.

In Nth field extractor the field is extracted by the position of the field , such as 0,1,2 . the position starts from 0. This is the most widely used extractor type for the flatfiles because it does not limit the size of the field as in the case of Fixed length extractor.

Delimited FlatFile :- In these kind of FlatFiles , the records are separated by some kind of delimiters such as newline,pipe(|),tab etc. The field extractor can be any of the two i.e fixed length or Nth field extractor. For Example :-

EmployeeId,EmployeeName,DOJ,City,Department
Employee,1234,QWERTY,12-08-2013,Banaglore,GHT
Employee,1235,PQRSTUV,12-08-2013,Banaglore,IMBT
Employee,1239,HOWAREU,19-08-2013,Banaglore,FGTH
Employee,1240,IMFINE,19-08-2013,Chennai,GCS
Employee,1236,HIHELLO,12-08-2013,Banaglore,QWER


Fixed Length:-In these kind of flatfiles , the length of the record is specified and it acts as the record delimiter , so after a particular size next record will begin , here also any of the two extractor types can be used. For Example :-

Employee1234Rakeshaaa12-08-2013BLRGCSEmployee1235Raghavend12-08-2013BLRGCSEmployee1239Ranjanqqq19-08-2013BLRGCSEmployee1240Satishxxx19-08-2013BLRGCSEmployee1236Sudhashuw12-08-2013BLRGCS

Variable length:- In these kind of flatfiles , each record is preceded by two bytes which indicate the size of the record which acts as the record delimiter, here also any of the two field extractor can be used. For example see the picture below :-

Here in the above picture you can see that the first two characters/bytes are used to denote the length of record which is acting as record delimiter.The ASCII value of the corresponding character is taken in account so in the above picture the length of first record is 32 as the ASCII value of NUL is 0 and of single space is 32 , similarly the length of second record is 37 as the ASCII value of NUL is 0 and of % is 37 . For inserting ASCII characters you can use the editor Notepad ++ , use this link http://notepad-plus-plus.org/ to get notepad ++ . In order to use the ASCII panel in notepad ++ go to Edit menu then select character panel .

Now the question is how to get the flatfiles into webMethods and parse it and convert it into document type.
To get flat files into webMethods you have to use either of the following :-
1. create a webMethods file polling port.
2. use the inbuilt service pub.file:getFile to get the file into webMethods.
3. If the file is present on any remote system then we have to get the files using the ftp and sftp services provided by webMethods . You can use the public service from the following folder "pub.client.ftp"
4. You can write your own custom service to get the file into webMethods .

I will discuss about all these above four points in my next blog , right now let us concentrate on after we get these files , how do we parse these files what are required ?

In order to parse the flat files we will be using the public service provided by webMethods which is available in the package "WmFlatFile" . The most used of them are as follows :-

So, In order to parse flat files in webMethods you need to create the corresponding structure of the falt file in webMethods . So, you have to create flat file schema in webMethods.

There is generally two ways to create the flat file schema in webMethods :-
1. First One is directly creating the record definition and field definition that are present in the flat file , if you are directly creating the schema then you have to be sure there is record identifier coming in our flat file, otherwise webMethods will be unable to parse the flat file .

2. Second one is via flat file dictionary , in flat file dictionary you create all the record and field definition that are being used in the flat file , then you can set this dictionary in the default record of the flat file schema . The advantage of this method is that the record identifier should nt be present in the flat file. Second it's a reusable component which can be used again and again.

Now we will proceed to the parsing of the flat files in webMethods , firstly we will have a look on how flat files are parsed and converted into IS Documents.

1. pub.flatFile:convertToValues :- This service converts flatfile data into IS document , the required inputs for this service is ffData(which is of Object type , it can take anything such as String,InputStream or Byte array) and ffSchema (fully qualified path of the flat file schema that you have created).

The output of this service is ffValues(IData)(It is the document that has been formed from your flat file , you can map it to your corresponding document type created for your flat file schema).
isValid(String) :- gives true/false accordingly if the flat file data is able to parsed successfully or unsuccessfully.
errors(IData Array) :- gets populated only when isValid is false
ffIterator(Object):- It is like a cursor which points to which all data has been parsed , populated when the iterate input parameter is kept true . It is generally used when we are dealing with large flat file data.

Complex Flat File Cases :-
Release Character :-  Suppose there is a case when you have to use the field delimiter specified in the flat file / or cases where the filed delimiter is present in the flat file. The release character enables you to use the field delimiter in the flat file.

For example suppose ,(comma) is the field delimiter and the comma is also used in some fields and while parsing it has to come in the field value . So, the comma should be preceded with the \(slash) , here slash will act as the release character. So, in the flat file where ever the \(slash) will be present before the ,(comma) it will not act as filed delimiter.
For release character there are two ways to specify it :-

1. Release Character (directly mentioned the release character here)
2. Character Position (mention the position where the release character will be present in the file).

Example of a release character based delimited file :-

Quoted Release Character :- Quoted release character is used to keep the section of a flat file as it is mentioned in the flat file , any delimiters included within the quoted release character will not be counted.

For quoted release character there are two ways to specify it :-

1. Character (directly mentioned the release character here)
2. Character Position (mention the position where the release character will be present in the file)

Note :- If you specify both release character and quoted release character , then the quoted release character is given preference while doing the flat file parsing.

Nested FlatFiles :- There are many case where you will find a record and the fields from the record will be referred in another record in the same flat file .
Please see the below example :-

Here in the above example you can see that the first record refers to the employee details , then in the next record we have the shift details of the employee.So, the Shift record can be considered as the nested record of the EMPLOYEE record. And in the last there is Total record which gives the total number of records present in the flat file.

The overall flat file structure looks like this :-

In the flat file definition , the new line is defined as the record delimiter and ,(comma) is defined as the field delimiter. After parsing the structure looks like as mentioned below :-

Enjoy Reading , If you have any Question related to webMethods please comment below , I will try to answer the questions. Please Subscribe for latest update on the blogs