All posts by Raj Nagalingam

Cinchoo – Running in hosted / UnitTest environment

Cinchoo discovers, opens and saves its own settings automatically in application folders. In hosted environments like IIS / MS-Test / NUnit etc, the base folder is protected from saving such changes. Cinchoo abnormally terminates the application with failure.

To overcome it, you must initialize Cinchoo configuration manger to the fully accessible folder at the application startup.

ChoConfigurationManager.OpenExeConfiguration(@”C:\Temp\test1.config”);

 

Cinchoo ETL – CSV Writer

Download source code

Download binary

Contents

1. Introduction

ChoETL is an open source ETL (extract, transform and load) framework for .NET. It is a code based library for extracting data from multiple sources, transforming, and loading into your very own data warehouse in .NET environment. You can have data in your data warehouse in no time.

This article talks about using CSVRWriter component offered by ChoETL framework. It is a simple utility class to save CSV data to a file.

UPDATE: Corresponding CSVReader article can be found here.

Features:

  • Follows CSV standard file rules. Gracefully handles data fields that contain commas and line breaks.
  • In addition to comma, most delimiting characters can be used, including tab delimited fields.
  • Supports culture specific date, currency and number formats while generating files.
  • Supports different character encoding.
  • Provides fine control of date, currency, enum, boolean, number formats when writing files.
  • Detailed and robust error handling, allowing you to quickly find and fix problems.
  • Shorten your development time.

2. Requirement

This framework library is written in C# using .NET 4.5 Framework.

3. “Hello World!” Sample

  • Open VS.NET 2013 or higher
  • Create a sample VS.NET (.NET Framework 4.5) Console Application project
  • Install ChoETL via Package Manager Console using Nuget Command: Install-Package ChoETL
  • Use the ChoETL namespace

Let’s begin by looking into a simple example of generating the below CSV file having 2 columns

Listing 3.1 Sample CSV data file

1,Tom
2,Carl
3,Mark

There are number of ways you can get the CSV file be created with minimal setup.

3.1. Quick write – Data First Approach

This is the zero-config and quickest way to create CSV file in no time. No typed POCO object is needed. Sample code below shows how to generate sample CSV file using dynamic objects

Listing 3.1.1 Write list of objects to CSV file

List<ExpandoObject> objs = new List<ExpandoObject>();
dynamic rec1 = new ExpandoObject();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
dynamic rec2 = new ExpandoObject();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
 
using (var parser = new ChoCSVWriter("Emp.csv"))
{
    parser.Write(objs);
}

In the above sample, we give the list of objects to CSVWriter at one pass to write them to CSV file.

Listing 3.1.2 Write each object to CSV file

using (var parser = new ChoCSVWriter("Emp.csv"))
{
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 1;
    rec1.Name = "Mark";
    parser.Write(item);

    dynamic rec1 = new ExpandoObject();
    rec1.Id = 2;
    rec1.Name = "Jason";
    parser.Write(item);
}

In the above sample, we take control of constructing, passing each and individual record to the CSVWriter to generate the CSV file using Write overload.

3.2. Code First Approach

This is another zeo-config way to generate CSV file using typed POCO class. First define a simple POCO class to match the underlying CSV file layout

Listing 3.2.1 Simple POCO entity class

public partial class EmployeeRecSimple
{
    public int Id { get; set; }
    public string Name { get; set; } 
}

In above, the POCO class defines two properties matching the sample CSV file template.

Listing 3.2.2 Saving to CSV file

List<EmployeeRecSimple> objs = new List<EmployeeRecSimple>();

EmployeeRecSimple rec1 = new EmployeeRecSimple();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
EmployeeRecSimple rec2 = new EmployeeRecSimple();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
 
using (var parser = new ChoCSVWriter<EmployeeRecSimple>("Emp.csv"))
{
    parser.Write(objs);
}

Above sample shows how to create CSV file from typed POCO class objects.

3.3. Configuration First Approach

In this model, we define the CSV configuration with all the necessary parameters along with CSV columns required to generate the sample CSV file.

Listing 3.3.1 Define CSV configuration

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.CSVRecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1));
config.CSVRecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));

In above, the class defines two CSV properties matching the sample CSV file template.

Listing 3.3.2 Generate CSV file without POCO object

List<ExpandoObject> objs = new List<ExpandoObject>();

dynamic rec1 = new ExpandoObject();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
dynamic rec2 = new ExpandoObject();
rec2.Id = 2;
rec2.Name = "Tom";
objs.Add(rec2);
 
using (var parser = new ChoCSVWriter("Emp.csv", config))
{
    parser.Write(objs);
}

The above sample code shows how to generate CSV file from list of dynamic objects using predefined CSV configuration setup. In the CSVWriter constructor, we specified the CSV configuration configuration object to obey the CSV layout schema while creating the file. If there are any mismatch in the name or count of CSV columns, will be reported as error and stops the writing process.

Listing 3.3.3 Saving CSV file with POCO object

List<EmployeeRecSimple> objs = new List<EmployeeRecSimple>();

EmployeeRecSimple rec1 = new EmployeeRecSimple();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
EmployeeRecSimple rec2 = new EmployeeRecSimple();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
 
using (var parser = new ChoCSVWriter<EmployeeRecSimple>("Emp.csv", config))
{
    parser.Write(objs);
}

Above sample code shows how to generate CSV file from list of POCO objects with CSV configuration object. In the CSVWriter constructor, we specified the CSV configuration configuration object.

3.4. Code First with declarative configuration

This is the combined approach to define POCO entity class along with attaching CSV configuration parameters declaratively. id is required column and name is optional value column with default value XXXX“. If name is not present, it will take the default value.

Listing 3.4.1 Define POCO Object

public class EmployeeRec
{
    [ChoCSVRecordField(1)]
    [Required]
    public int? Id
    {
        get;
        set;
    }

    [ChoCSVRecordField(2)]
    [DefaultValue("XXXX")]
    public string Name
    {
        get;
        set;
    }

    public override string ToString()
    {
        return "{0}. {1}.".FormatString(Id, Name);
    }
}

The code above illustrates about defining POCO object with nessasary attributes required to generate CSV file. First thing defines property for each record field with ChoCSVRecordFieldAttribute to qualify for CSV record mapping. Each property must specify position in order to be mapped to CSV column. Position is 1 based. Id is a required property. We decorated it with RequiredAttribute. Name is given default value using DefaultValueAttribute. It means that if the Name value is not set in the object, CSVWriter spits the default value ‘XXXX’ to the file.

It is very simple and ready to save CSV data in no time.

Listing 3.4.2 Saving CSV file with POCO object

List<EmployeeRec> objs = new List<EmployeeRec>();

EmployeeRec rec1 = new EmployeeRec();
rec1.Id = 10;
rec1.Name = "Mark";
objs.Add(rec1);
 
EmployeeRec rec2 = new EmployeeRec();
rec2.Id = 200;
rec2.Name = "Lou";
objs.Add(rec2);
 
using (var parser = new ChoCSVWriter<EmployeeRec>("Emp.csv"))
{
    parser.Write(objs);
}

We start by creating a new instance of ChoCSVWriter object. That’s all. All the heavy lifting of genering CSV data from the objects is done by the writer under the hood.

By default, CSVWriter discovers and uses default configuration parameters while saving CSV file. These can be overridable according to your needs. The following sections will give you in-depth details about each configuration attributes.

4. Writing All Records

It is as easy as setting up POCO object match up with CSV file structure, construct the list of objects and pass it to CSVWriter’s Write method. This will write the entire list of objects into CSV file in one single call.

Listing 4.1 Write to CSV File

List<EmployeeRec> objs = new List<EmployeeRec>();
//Construct and attach objects to this list
...

using (var parser = new ChoCSVWriter<EmployeeRec>("Emp.csv"))
{
    parser.Write(objs);
}

or:

Listing 4.2 Writer to CSV file stream

List<EmployeeRec> objs = new List<EmployeeRec>();
//Construct and attach objects to this list
...

using (var tx = File.OpenWrite("Emp.csv"))
{
    using (var parser = new ChoCSVWriter<EmployeeRec>(tx))
    {
        parser.Write(objs);
    }
}

This model keeps your code elegant, clean, easy to read and maintain.

5. Write Records Manually

This is an alternative way to write each and individual record to CSV file in case when the POCO objects are constructed in a disconnected way.

Listing 5.1 Wrting to CSV file

var writer = new ChoCSVWriter<EmployeeRec>("Emp.csv");

EmployeeRec rec1 = new EmployeeRec();
rec1.Id = 10;
rec1.Name = "Mark";
 
writer.Write(rec1);

EmployeeRec rec2 = new EmployeeRec();
rec1.Id = 11;
rec1.Name = "Top"; 

writer.Write(rec2);

6. Customize CSV Record

Using ChoCSVRecordObjectAttribute, you can customize the POCO entity object declaratively.

Listing 6.1 Customizing POCO object for each record

[ChoCSVRecordObject(Encoding = "Encoding.UTF32", 
ErrorMode = ChoErrorMode.IgnoreAndContinue, IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All)]
public class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "id")]
    public int Id { get; set; }
    [ChoCSVRecordField(2, FieldName ="Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available attributes to carry out customization of CSV load operation on a file.

  • Delimiter – The value used to separate the fields in a CSV row. Default is Culture.TextInfo.ListSeparator used.
  • EOLDelimiter – The value used to separate CSV rows. Default is \r\n (NewLine).
  • Culture – The culture info used to read and write.
  • IgnoreEmptyLine – N/A.
  • Comments – N/A.
  • QuoteChar – The value used to escape fields that contain a delimiter, quote, or line ending.
  • QuoteAllFields – A flag that tells the writer whether all fields written should have quotes around them; regardless if the field contains anything that should be escaped.
  • Encoding – The encoding of the CSV file.
  • HasExcelSeperator – A flag that tells the writer to spit out the excel seperator information in the out file.
  • ColumnCountStrict – This flag indicates if an exception should be thrown if CSV field configuration mismatch with the data object members.
  • ColumnOrderStrict – N/A.
  • BufferSize – The size of the internal buffer that is used when reader is from the StreamWriter.
  • ErrorMode – This flag indicates if an exception should be thrown if writing and an expected field is failed to write. This can be overridden per property. Possible values are:
    • IgnoreAndContinue – Ignore the error, record will be skipped and continue with next.
    • ReportAndContinue – Report the error to POCO entity if it is of IChoNotifyRecordWrite type
    • ThrowAndStop – Throw the error and stop the execution
  • IgnoreFieldValueMode – N/A.
  • ObjectValidationMode – A flag to let the reader know about the type of validation to be performed with record object. Possible values are:
    • Off – No object validation performed. (Default)
    • MemberLevel – Validation performed before each CSV property gets written to the file.
    • ObjectLevel – Validation performed before all the POCO properties are written to the file.

7. Customize CSV Header

By attaching ChoCSVFileHeaderAttribute to POCO entity object declaratively, you can influence the writer to generate CSV header when creating CSV file.

Listing 6.1 Customizing POCO object for file header

[ChoCSVFileHeader]
public class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "id")]
    public int Id { get; set; }
    [ChoCSVRecordField(2, FieldName ="Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available members to add some customization to it according to your need.

  • FillChar – Padding character used when size of the CSV column header is short of the column size (ChoCSVRecordFieldAttribute.Size or ChoCSVRecordFieldConfiguration.Size). Default is ‘\0’, padding will be off.
  • Justification – Column header alignment. Default is Left.
  • TrimOption – N/A.
  • Truncate – This flag tells that the writer to truncate the CSV column header value if it over the column size. Default is false.

8. Customize CSV Fields

For each CSV column, you can specify the mapping in POCO entity property using ChoCSVRecordFieldAttribute.

Listing 6.1 Customizing POCO object for CSV columns

[ChoCSVFileHeader]
public class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "id")]
    public int Id { get; set; }
    [ChoCSVRecordField(2, FieldName ="Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available members to add some customization to it for each property:

  • FieldPosition – When mapping by position, you specify the index of the CSV column that you want to use for that property. It is 1 based.
  • FieldName – CSV Column name header. If not specified, POCO object property name will be used as column header.
  • FillChar – Padding character used when size of the CSV column value is short of the column size. Default is ‘\0’, padding will be off.
  • FieldValueJustification – Column value alignment. Default is Left.
  • FieldValueTrimOption – N/A.
  • Truncate – This flag tells that the writer to truncate the CSV column value if it over the column size. Default is false.
  • Size – Size of CSV column value.
  • QuoteField – A flag that tells the writer that the CSV column value is surrounded by quotes.
  • ErrorMode – This flag indicates if an exception should be thrown if writing and an expected field failed to convert and write. Possible values are:
    • IgnoreAndContinue – Ignore the error and continue to load other properties of the record.
    • ReportAndContinue – Report the error to POCO entity if it is of IChoRecord type.
    • ThrowAndStop – Throw the error and stop the execution.
  • IgnoreFieldValueMode – N/A

8.1. DefaultValue

Any POCO entity property can be specified with default value using System.ComponentModel.DefaultValueAttribute. It is the value used to write when the CSV value null (controlled via IgnoreFieldValueMode).

8.2. ChoFallbackValue

Any POCO entity property can be specified with fallback value using ChoETL.ChoFallbackValueAttribute. It is the value used and set to the property when the CSV value failed to convert as text. Fallback value only set when ErrorMode is either IgnoreAndContinue or ReportAndContinue.

8.3. Type Converters

Most of the primitive types are automatically converted to string/text and save them to CSV file. If the value of the CSV field aren’t automatically be converted into the text value, you can specify a custom / built-in .NET converters to convert the value to text. These can be either IValueConverter or TypeConverter converters.

The methods to use to convert/format property values to text are IValueConverter.ConvertBack() or TypeConvert.ConvertTo().

Listing 8.3.1 Specifying type converters

[ChoCSVFileHeader]
public class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    public int Id { get; set; }
    [ChoCSVRecordField(2, FieldName ="Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Listing 8.3.2 IntConverter implementation

public class IntConverter : IValueConverter
{
    public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
    {
        return value;
    }
 
    public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
    {
        int intValue = (int)value;
        return intValue.ToString("D4");
    }
}

In the example above, we defined custom IntConverter class. And showed how to format ‘IdCSV property with leading zeros.

8.4. Validations

CSVWriter leverages both System.ComponentModel.DataAnnotations and Validation Block validation attributes to specify validation rules for individual fields of POCO entity. Refer to the MSDN site for a list of available DataAnnotations validation attributes.

Listing 8.4.1 Using validation attributes in POCO entity

[ChoCSVFileHeader]
[ChoCSVRecordObject(Encoding = "Encoding.UTF32", ErrorMode = ChoErrorMode.IgnoreAndContinue,
       IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, ThrowAndStopOnMissingField = false)]
public partial class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
 
    [ChoCSVRecordField(2, FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
}

In example above, used Range validation attribute for Id property. Required validation attribute to Name property. CSVWriter performs validation on them before saving the data to file when Configuration.ObjectValidationMode is set to ChoObjectValidationMode.MemberLevel or ChoObjectValidationMode.ObjectLevel.

Some cases, you may want to take control and perform manual self validation within the POCO entity class. This can be achieved by inheriting POCO object from IChoValidatable interface.

Listing 8.4.2 Manual validation on POCO entity

[ChoCSVFileHeader]
[ChoCSVRecordObject(Encoding = "Encoding.UTF32", ErrorMode = ChoErrorMode.IgnoreAndContinue,
       IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, ThrowAndStopOnMissingField = false)]
public partial class EmployeeRec : IChoValidatable
{
    [ChoCSVRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
 
    [ChoCSVRecordField(2, FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool TryValidate(object target, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public bool TryValidateFor(object target, string memberName, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public void Validate(object target)
    {
    }
 
    public void ValidateFor(object target, string memberName)
    {
    }
}

Sample above shows how to implement custom self-validation in POCO object.

IChoValidatable interface exposes below methods

  • TryValidate – Validate entire object, return true if all validation passed. Otherwise return false.
  • Validate – Validate entire object, throw exception if validation is not passed.
  • TryValidateFor – Validate specific property of the object, return true if all validation passed. Otherwise return false.
  • ValidateFor – Validate specific property of the object, throw exception if validation is not passed.

9. Excel Field Separator

By setting HasExcelSeperator declaratively on POCO object or ChoCSVRecordConfiguration.HasExcelSeperator to true to generate excel field separator in the data file.

Listing 9.1 Specifying HasExcelSeperator to POCO object declaratively

[ChoCSVFileHeader]
[ChoCSVRecordObject(HasExcelSeparator = true)]
public class EmployeeRec
{
    [ChoCSVRecordField(1)]
    [Required]
    [ChoFallbackValue(100)]
    [Range(100, 10000)]
    public int? Id
    {
        get;
        set;
    }
    [ChoCSVRecordField(2)]
    [DefaultValue("XXXX")]
    public string Name
    {
        get;
        set;
    }
 
    public override string ToString()
    {
        return "{0}. {1}.".FormatString(Id, Name);
    }
}

Listing 9.2 Specifying HasExcelSeperator via configuration

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.CSVRecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1));
config.CSVRecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));
config.HasExcelSeparator = true;
 
List<EmployeeRecSimple> objs = new List<EmployeeRecSimple>();
EmployeeRecSimple rec1 = new EmployeeRecSimple();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
EmployeeRecSimple rec2 = new EmployeeRecSimple();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
 
using (var parser = new ChoCSVWriter<EmployeeRecSimple>("Emp.csv", config))
{
    parser.Write(objs);
}

Listing 9.3 Sample CSV file with Excel field separator

sep=,
1,Mark 
2,Jason

10. Callback Mechanism

CSVWriter offers industry standard CSV data file generation out of the box to handle most of the needs. If the generation process is not handling any of your needs, you can use the callback mechanism offered by CSVWriter to handle such situations. In order to participate in the callback mechanism, Either POCO entity object or DataAnnotation’s MetadataType type object must be inherited by IChoNotifyRecordWrite interface.

Tip: Any exceptions raised out of these interface methods will be ignored.

IChoNotifyRecordWrite exposes the below methods:

  • BeginWrite – Invoked at the begin of the CSV file write
  • EndWrite – Invoked at the end of the CSV file write
  • BeforeRecordWrite – Raised before the CSV record write
  • AfterRecordWrite – Raised after CSV record write
  • RecordWriteError – Raised when CSV record errors out while writing
  • BeforeRecordFieldWrite – Raised before CSV column value write
  • AfterRecordFieldWrite – Raised after CSV column value write
  • RecordFieldWriteError – Raised when CSV column value errors out while writing

Listing 10.1 Direct POCO callback mechanism implementation

[ChoCSVFileHeader]
[ChoCSVRecordObject(Encoding = "Encoding.UTF32", ErrorMode = ChoErrorMode.IgnoreAndContinue,
       IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, ThrowAndStopOnMissingField = false)]
public partial class EmployeeRec : IChoNotifyrRecordWrite
{
    [ChoCSVRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
    
    [ChoCSVRecordField(2, FieldName = "Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }

    public bool AfterRecordFieldWrite(object target, int index, string propName, object value)
    {
        throw new NotImplementedException();
    }

    public bool AfterRecordWrite(object target, int index, object source)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordFieldWrite(object target, int index, string propName, ref object value)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordWrite(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }

    public bool BeginWrite(object source)
    {
        throw new NotImplementedException();
    }

    public void EndWrite(object source)
    {
        throw new NotImplementedException();
    }

    public bool RecordFieldWriteError(object target, int index, string propName, object value, Exception ex)
    {
        throw new NotImplementedException();
    }

    public bool RecordWriteError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
}

Listing 10.2 MetaDataType based callback mechanism implementation

[ChoCSVFileHeader]
[ChoCSVRecordObject(Encoding = "Encoding.UTF32", ErrorMode = ChoErrorMode.IgnoreAndContinue,
       IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, ThrowAndStopOnMissingField = false)]
public class EmployeeRecMeta : IChoNotifyRecordWrite
{
    [ChoCSVRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }

    [ChoCSVRecordField(2, FieldName = "Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordFieldWrite(object target, int index, string propName, object value)
    {
        throw new NotImplementedException();
    }

    public bool AfterRecordWrite(object target, int index, object source)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordFieldWrite(object target, int index, string propName, ref object value)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordWrite(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }

    public bool BeginWrite(object source)
    {
        throw new NotImplementedException();
    }

    public void EndWrite(object source)
    {
        throw new NotImplementedException();
    }

    public bool RecordFieldWriteError(object target, int index, string propName, object value, Exception ex)
    {
        throw new NotImplementedException();
    }

    public bool RecordWriteError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
} 

[MetadataType(typeof(EmployeeRecMeta))]
public partial class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
    
    [ChoCSVRecordField(2, FieldName = "Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
}

10.1 BeginWrite

This callback invoked once at the beginning of the CSV file write. source is the CSV file stream object. In here you have chance to inspect the stream, return true to continue the CSV generation. Return false to stop the generation.

Listing 10.1.1 BeginWrite Callback Sample

public bool BeginWrite(object source)
{
    StreamReader sr = source as StreamReader;
    return true;
}

10.2 EndWrite

This callback invoked once at the end of the CSV file generation. source is the CSV file stream object. In here you have chance to inspect the stream, do any post steps to be performed on the stream.

Listing 10.2.1 EndWrite Callback Sample

public void EndWrite(object source)
{
    StreamReader sr = source as StreamReader;
}

10.3 BeforeRecordWrite

This callback invoked before each POCO record object is written to CSV file. target is the instance of the POCO record object. index is the line index in the file. source is the CSV record line. In here you have chance to inspect the POCO object, and generate the CSV record line if needed.

Tip: If you want to skip the record from writing, set the source to null.

Tip: If you want to take control of CSV record line generation, set the source to valid CSV record line text. 

Return true to continue the load process, otherwise return false to stop the process.

Listing 10.3.1 BeforeRecordWrite Callback Sample

public bool BeforeRecordWrite(object target, int index, ref object source)
{
    source = "1,Raj";
    return true;
}

10.4 AfterRecordWrite

This callback invoked after each POCO record object is written to CSV file. target is the instance of the POCO record object. index is the line index in the file. source is the CSV record line. In here you have chance to do any post step operation with the record line.

Return true to continue the load process, otherwise return false to stop the process.

Listing 10.4.1 AfterRecordWrite Callback Sample

public bool AfterRecordWrite(object target, int index, object source)
{
    string line = source as string;
    return true;
}

10.5 RecordWriteError

This callback invoked if error encountered while writing POCO record object. target is the instance of the POCO record object. index is the line index in the file. source is the CSV record line. ex is the exception object. In here you have chance to handle the exception. This method invoked only when Configuration.ErrorMode is ReportAndContinue.

Return true to continue the load process, otherwise return false to stop the process.

Listing 10.5.1 RecordWriteError Callback Sample

public bool RecordLoadError(object target, int index, object source, Exception ex)
{
    string line = source as string;
    return true;
}

10.6 BeforeRecordFieldWrite

This callback invoked before each CSV record column is written to CSV file. target is the instance of the POCO record object. index is the line index in the file. propName is the CSV record property name. value is the CSV column value. In here, you have chance to inspect the CSV record property value and perform any custom validations etc.

Return true to continue the load process, otherwise return false to stop the process.

Listing 10.6.1 BeforeRecordFieldWrite Callback Sample

public bool BeforeRecordFieldWrite(object target, int index, string propName, ref object value)
{
    return true;
}

10.7 AfterRecordFieldWrite

This callback invoked after each CSV record column value is written to CSV file. target is the instance of the POCO record object. index is the line index in the file. propName is the CSV record property name. value is the CSV column value. Any post field operation can be performed here, like computing other properties, validations etc.

Return true to continue the load process, otherwise return false to stop the process.

Listing 10.7.1 AfterRecordFieldWrite Callback Sample

public bool AfterRecordFieldWrite(object target, int index, string propName, object value)
{
    return true;
}

10.8 RecordWriteFieldError

This callback invoked when error encountered while writing CSV record column value. target is the instance of the POCO record object. index is the line index in the file. propName is the CSV record property name. value is the CSV column value. ex is the exception object. In here you have chance to handle the exception. This method invoked only after the below two sequences of steps performed by the CSVReader

  • CSVWriter looks for FallbackValue value of each CSV property. If present, it tries to use it to write.
  • If the FallbackValue value not present and the Configuration.ErrorMode is specified as ReportAndContinue., this callback will be executed.

Return true to continue the load process, otherwise return false to stop the process.

Listing 10.8.1 RecordFieldWriteError Callback Sample

public bool RecordFieldWriteError(object target, int index, string propName, object value, Exception ex)
{
    return true;
}

11. Customization

CSVWriter automatically detects and loads the configuration settings from POCO entity. At runtime, you can customize and tweak these parameters before CSV generation. CSVWriter exposes Configuration property, it is of ChoCSVRecordConfiguration object. Using this property, you can perform the customization.

Listing 11.1 Customizing CSVWriter at run-time

class Program
{
    static void Main(string[] args)
    {        
        List<ExpandoObject> objs = new List<ExpandoObject>();
        dynamic rec1 = new ExpandoObject();
        rec1.Id = 1;
        rec1.Name = "Mark";
        objs.Add(rec1);

        dynamic rec2 = new ExpandoObject();
        rec2.Id = 2;
        rec2.Name = "Jason";
        objs.Add(rec2);

        using (var parser = new ChoCSVWriter("Emp.csv"))
        {
            parser.Configuration.ColumnCountStrict = true;
            parser.Write(objs);
        }
    }
}

12. Using Dynamic Object

So far, the article explained about using CSVWriter with POCO object. CSVWriter also supports generating CSV file without POCO entity objects It leverages .NET dynamic feature. The sample below shows how to generate CSV stream using dynamic objects. The CSV schema is determined from first object. If there is mismatch found in the dynamic objects member values, error will be raised and stop the generation process.

The sample below shows it:

Listing 12.1 Generating CSV file from dynamic objects

class Program
{
    static void Main(string[] args)
    {        
        List<ExpandoObject> objs = new List<ExpandoObject>();
        dynamic rec1 = new ExpandoObject();
        rec1.Id = 1;
        rec1.Name = "Mark";
        objs.Add(rec1);

        dynamic rec2 = new ExpandoObject();
        rec2.Id = 2;
        rec2.Name = "Jason";
        objs.Add(rec2);

        using (var parser = new ChoCSVWriter("Emp.csv"))
        {
            parser.Configuration.ColumnCountStrict = true;
            parser.Write(objs);
        }
    }
}

13. Exceptions

CSVReader throws different types of exceptions in different situations.

  • ChoParserException – CSV file is bad and parser not able to recover.
  • ChoRecordConfigurationException – Any invalid configuration settings are specified, this exception will be raised.
  • ChoMissingRecordFieldException – A property is missing for a CSV column, this exception will be raised.

14. Tips

CSVWriter automatically quote the column values if either of the following conditions met

  • the value contains newline / delimiter characters
  • the value contains quote character
  • the value contains leading or trailing spaces

Other situations, if you want to add quotes around values, it can be specified in QuoteField parameter as true.

Listing 14.1.1 Multiline column values in CSV file

[ChoCSVFileHeader]
[ChoCSVRecordObject(HasExcelSeparator = true)]
public class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "NewId")]
    [Required]
    [ChoFallbackValue(100)]
    [Range(100, 10000)]
    public int? Id
    {
        get;
        set;
    }
    [ChoCSVRecordField(2, QuoteField = true)]
    [DefaultValue("XXXX")]
    public string Name
    {
        get;
        set;
    }
 
    public override string ToString()
    {
        return "{0}. {1}.".FormatString(Id, Name);
    }
}

15. Using MetadataType Annotation

Cinchoo ETL works better with data annotation’s MetadataType model. It is way to attach MetaData class to data model class. In this associated class, you provide additional metadata information that is not in the data model. It roles is to add attribute to a class without having to modify this one. You can add this attribute that takes a single parameter to a class that will have all the attributes. This is useful when the POCO classes are auto generated (by Entity Framework, MVC etc) by an automatic tools. This is why second class come into play. You can add new stuffs without touching the generated file. Also this promotes modularization by separating the concerns into multiple classes.

For more information about it, please search in MSDN.

Listing 15.1 MetadataType annotation usage sample

[MetadataType(typeof(EmployeeRecMeta))]
public class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; }
}

[ChoCSVFileHeader]
[ChoCSVRecordObject(Encoding = "Encoding.UTF32", ErrorMode = ChoErrorMode.ThrowAndStop,
IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, ThrowAndStopOnMissingField = false, 
    ObjectValidationMode = ChoObjectValidationMode.MemberLevel)]
public class EmployeeRecMeta : IChoNotifyRecordWrite, IChoValidatable
{
    [ChoCSVRecordField(1, FieldName = "id", ErrorMode = ChoErrorMode.ReportAndContinue )]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, 1, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }

    [ChoCSVRecordField(2, FieldName = "Name", QuoteField = true)]
    [StringLength(1)]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
    public bool AfterRecordFieldWrite(object target, int index, string propName, object value)
    {
        throw new NotImplementedException();
    }

    public bool AfterRecordWrite(object target, int index, object source)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordFieldWrite(object target, int index, string propName, ref object value)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordWrite(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }

    public bool BeginWrite(object source)
    {
        throw new NotImplementedException();
    }

    public void EndWrite(object source)
    {
        throw new NotImplementedException();
    }

    public bool RecordFieldWriteError(object target, int index, string propName, object value, Exception ex)
    {
        throw new NotImplementedException();
    }

    public bool RecordWriteError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
 
    public bool TryValidate(object target, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public bool TryValidateFor(object target, string memberName, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public void Validate(object target)
    {
    }
 
    public void ValidateFor(object target, string memberName)
    {
    }
}

In above EmployeeRec is the data class. Contains only domain specific properties and operations. Mark it very simple class to look at it.

We separate the validation, callback mechanism, configuration etc into metadata type class, EmployeeRecMeta.

16. Configuration Choices

If the POCO entity class is an auto-generated class or exposed via library or it is a sealed class, it limits you to attach CSV schema definition to it declaratively. In such case, you can choose one of the options below to specify CSV layout configuration

  • Manual Configuration
  • Auto Map Configuration
  • Attaching MetadataType class

I’m going to show you how to configure the below POCO entity class on each approach

Listing 16.1 Sealed POCO entity class

public sealed class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; }
}

16.1 Manual Configuration

Define a brand new configuration object from scratch and add all the necessary CSV fields to the ChoCSVConfiguration.CSVRecordFieldConfigurations collection property. This option gives you greater flexibility to control the configuration of CSV parsing. But the downside is that possibility of making mistakes and hard to manage them if the CSV file layout is large,

Listing 16.1.1 Manual Configuration

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.CSVFileHeaderConfiguration.HasHeaderRecord = true;
config.ThrowAndStopOnMissingField = true;
config.CSVRecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1));
config.CSVRecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));

16.2 Auto Map Configuration

This is an alternative approach and very less error-prone method to auto map the CSV columns for the POCO entity class.

First define a schema class for EmployeeRec POCO entity class as below

Listing 16.2.1 Auto Map class

public class EmployeeRecMap
{
    [ChoCSVRecordField(1, FieldName = "id")]
    public int Id { get; set; }
 
    [ChoCSVRecordField(2, FieldName = "Name")]
    public string Name { get; set; } 
}

Then you can use it to auto map CSV columns by using ChoCSVRecordConfiguration.MapRecordFields method

Listing 16.2.2 Using Auto Map configuration

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.MapRecordFields<EmployeeRecMap>();

foreach (var e in new ChoCSVReader<EmployeeRec>("Emp.csv", config)) 
    Console.WriteLine(e.ToString());

16.3 Attaching MetadataType class

This is one another approach to attach MetadataType class for POCO entity object. Previous approach simple care for auto mapping of CSV columns only. Other configuration properties like property converters, parser parameters, default/fallback values etc. are not considered.

This model, accounts for everything by defining MetadataType class and specifying the CSV configuration parameters declaratively. This is useful when your POCO entity is sealed and not partial class. Also it is one of favorable and less error-prone approach to configure CSV parsing of POCO entity.

Listing 16.3.1 Define MetadataType class

[ChoCSVFileHeader()]
[ChoCSVRecordObject(Encoding = "Encoding.UTF32", ErrorMode = ChoErrorMode.ReportAndContinue,
IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, ThrowAndStopOnMissingField = false, 
    ObjectValidationMode = ChoObjectValidationMode.MemberLevel)]
public class EmployeeRecMeta : IChoNotifyRecordWrite, IChoValidatable
{
    [ChoCSVRecordField(1, FieldName = "id", ErrorMode = ChoErrorMode.ReportAndContinue )]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, 1, ErrorMessage = "Id must be > 0.")]
    //[ChoFallbackValue(1)]
    public int Id { get; set; }
    [ChoCSVRecordField(2, FieldName = "Name", QuoteField = true)]
    [StringLength(1)]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
    public bool AfterRecordFieldWrite(object target, int index, string propName, object value)
    {
        throw new NotImplementedException();
    }

    public bool AfterRecordWrite(object target, int index, object source)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordFieldWrite(object target, int index, string propName, ref object value)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordWrite(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }

    public bool BeginWrite(object source)
    {
        throw new NotImplementedException();
    }

    public void EndWrite(object source)
    {
        throw new NotImplementedException();
    }

    public bool RecordFieldWriteError(object target, int index, string propName, object value, Exception ex)
    {
        throw new NotImplementedException();
    }

    public bool RecordWriteError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
 
    public bool TryValidate(object target, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public bool TryValidateFor(object target, string memberName, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public void Validate(object target)
    {
    }
 
    public void ValidateFor(object target, string memberName)
    {
    }
}

Listing 16.3.2 Attaching MetadataType class

//Attach metadata 
ChoMetadataObjectCache.Default.Attach<EmployeeRec>(new EmployeeRecMeta());

using (var tx = File.OpenWrite("Emp.csv"))
{
    using (var parser = new ChoCSVWriter<EmployeeRec>(tx))
    {
        parser.Write(objs);
    }
}

17. ToText Helper Method

This is little nifty helper method to generate CSV formatted output from list of objects. It helps you to run and play with different options to see the CSV output quickly in test environment.

static void ToTextTest()
{
    List<EmployeeRec> objs = new List<EmployeeRec>();
    EmployeeRec rec1 = new EmployeeRec();
    rec1.Id = 10;
    rec1.Name = "Mark";
    objs.Add(rec1);
 
    EmployeeRec rec2 = new EmployeeRec();
    rec2.Id = 200;
    rec2.Name = "Lou";
    objs.Add(rec2);
 
    Console.WriteLine(ChoCSVWriter.ToText(objs));
}

18. Writing DataReader Helper Method

This helper method lets you to create CSV file / stream from ADO.NET DataReader.

static void WriteDataReaderTest()
{
    string connString = @"Data Source=(localdb)\v11.0;Initial Catalog=TestDb;Integrated Security=True";
 
    SqlConnection conn = new SqlConnection(connString);
    conn.Open();
    SqlCommand cmd = new SqlCommand("SELECT * FROM Members", conn);
    IDataReader dr = cmd.ExecuteReader();
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer, config))
    {
        parser.Write(dr);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

19. Writing DataTable Helper Method

This helper method lets you to create CSV file / stream from ADO.NET DataTable.

static void WriteDataTableTest()
{
    string connString = @"Data Source=(localdb)\v11.0;Initial Catalog=TestDb;Integrated Security=True";

    SqlConnection conn = new SqlConnection(connString);
    conn.Open();
    SqlCommand cmd = new SqlCommand("SELECT * FROM Members", conn);
    SqlDataAdapter da = new SqlDataAdapter(cmd);
    DataTable dt = new DataTable();
    da.Fill(dt);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer, config))
    {
        parser.Write(dt);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

20. Advanced Topics

20.1 Override Converters Format Specs

Cinchoo ETL automatically parses and converts each CSV column values to the corresponding CSV column’s underlying data type seamlessly. Most of the basic .NET types are handled automatically without any setup needed.

This is achieved through two key settings in the ETL system

  1. ChoCSVRecordConfiguration.CultureInfo – Represents information about a specific culture including the names of the culture, the writing system, and the calendar used, as well as access to culture-specific objects that provide information for common operations, such as formatting dates and sorting strings. Default is ‘en-US’.
  2. ChoTypeConverterFormatSpec – It is global format specifier class holds all the intrinsic .NET types formatting specs.

In this section, I’m going to talk about changing the default format specs for each .NET intrinsic data types according to parsing needs.

ChoTypeConverterFormatSpec is singleton class, the instance is exposed via ‘Instance’ static member. It is thread local, means that there will be separate instance copy kept on each thread.

There are 2 sets of format specs members given to each intrinsic type, one for loading and another one for writing the value, except for Boolean, Enum, DataTime types. These types have only one member for both loading and writing operations.

Specifying each intrinsic data type format specs through ChoTypeConverterFormatSpec will impact system wide. ie. By setting ChoTypeConverterFormatSpec.IntNumberStyle = NumberStyles.AllowParentheses, will impact all integer members of CSV objects to allow parentheses. If you want to override this behavior and take control of specific CSV data member to handle its own unique parsing of CSV value from global system wide setting, it can be done by specifying TypeConverter at the CSV field member level. Refer section 13.4 for more information.

Listing 20.1.1 ChoTypeConverterFormatSpec Members

public class ChoTypeConverterFormatSpec
{
    public static readonly ThreadLocal<ChoTypeConverterFormatSpec> Instance = new ThreadLocal<ChoTypeConverterFormatSpec>(() => new ChoTypeConverterFormatSpec());
 
    public string DateTimeFormat { get; set; }
    public ChoBooleanFormatSpec BooleanFormat { get; set; }
    public ChoEnumFormatSpec EnumFormat { get; set; }
 
    public NumberStyles? CurrencyNumberStyle { get; set; }
    public string CurrencyFormat { get; set; }
 
    public NumberStyles? BigIntegerNumberStyle { get; set; }
    public string BigIntegerFormat { get; set; }
 
    public NumberStyles? ByteNumberStyle { get; set; }
    public string ByteFormat { get; set; }
 
    public NumberStyles? SByteNumberStyle { get; set; }
    public string SByteFormat { get; set; }
 
    public NumberStyles? DecimalNumberStyle { get; set; }
    public string DecimalFormat { get; set; }
 
    public NumberStyles? DoubleNumberStyle { get; set; }
    public string DoubleFormat { get; set; }
 
    public NumberStyles? FloatNumberStyle { get; set; }
    public string FloatFormat { get; set; }
 
    public string IntFormat { get; set; }
    public NumberStyles? IntNumberStyle { get; set; }
 
    public string UIntFormat { get; set; }
    public NumberStyles? UIntNumberStyle { get; set; }
 
    public NumberStyles? LongNumberStyle { get; set; }
    public string LongFormat { get; set; }
 
    public NumberStyles? ULongNumberStyle { get; set; }
    public string ULongFormat { get; set; }
 
    public NumberStyles? ShortNumberStyle { get; set; }
    public string ShortFormat { get; set; }
 
    public NumberStyles? UShortNumberStyle { get; set; }
    public string UShortFormat { get; set; }
}

Sample below shows how to load CSV data stream having ‘se-SE’ (Swedish) culture specific data using CSVReader. Also the input feed comes with ‘EmployeeNo’ values containing parentheses. In order to make the load successful, we have to set the ChoTypeConverterFormatSpec.IntNumberStyle to NumberStyles.AllowParenthesis.

Listing 20.1.2 Using ChoTypeConverterFormatSpec in code

static void FormatSpecDynamicTest()
{
    ChoTypeConverterFormatSpec.Instance.DateTimeFormat = "d";
    ChoTypeConverterFormatSpec.Instance.BooleanFormat = ChoBooleanFormatSpec.YOrN;
 
    List<ExpandoObject> objs = new List<ExpandoObject>();
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 10;
    rec1.Name = "Mark";
    rec1.JoinedDate = new DateTime(2001, 2, 2);
    rec1.IsActive = true;
    objs.Add(rec1);
 
    dynamic rec2 = new ExpandoObject();
    rec2.Id = 200;
    rec2.Name = "Lou";
    rec2.JoinedDate = new DateTime(1990, 10, 23);
    rec2.IsActive = false;
    objs.Add(rec2);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer).WithFirstLineHeader())
    {
        parser.Write(objs);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

20.2 Currency Support

Cinchoo ETL provides ChoCurrency object to read and write currency values in CSV files. ChoCurrency is a wrapper class to hold the currency value in decimal type along with support of serializing them in text format during CSV load.

Listing 20.2.1 Using Currency members in dynamic model

static void CurrencyDynamicTest()
{
    ChoTypeConverterFormatSpec.Instance.CurrencyFormat = "C2";
 
    List<ExpandoObject> objs = new List<ExpandoObject>();
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 10;
    rec1.Name = "Mark";
    rec1.JoinedDate = new DateTime(2001, 2, 2);
    rec1.IsActive = true;
    rec1.Salary = new ChoCurrency(100000);
    objs.Add(rec1);
 
    dynamic rec2 = new ExpandoObject();
    rec2.Id = 200;
    rec2.Name = "Lou";
    rec2.JoinedDate = new DateTime(1990, 10, 23);
    rec2.IsActive = false;
    rec2.Salary = new ChoCurrency(150000);
    objs.Add(rec2);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer).WithFirstLineHeader().QuoteAllFields())
    {
        parser.Write(objs);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

Sample above shows how to output currency values using dynamic object model. As the currency output will have thousand comma separator, this will fail to generate CSV file. To overcome this issue, we specify the writer to quote all fields.

PS: The format of the currency value is figured by CSVReader through ChoRecordConfiguration.Culture and ChoTypeConverterFormatSpec.CurrencyFormat.

Sample below shows how to use ChoCurrency CSV field in POCO entity class.

Listing 20.2.2 Using Currency members in POCO model

public class EmployeeRecWithCurrency
{
    public int Id { get; set; }
    public string Name { get; set; }
    public ChoCurrency Salary { get; set; }
}
 
static void CurrencyPOCOTest()
{
    List<EmployeeRecWithCurrency> objs = new List<EmployeeRecWithCurrency>();
    EmployeeRecWithCurrency rec1 = new EmployeeRecWithCurrency();
    rec1.Id = 10;
    rec1.Name = "Mark";
    rec1.Salary = new ChoCurrency(100000);
    objs.Add(rec1);
 
    EmployeeRecWithCurrency rec2 = new EmployeeRecWithCurrency();
    rec2.Id = 200;
    rec2.Name = "Lou";
    rec2.Salary = new ChoCurrency(150000);
    objs.Add(rec2);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter<EmployeeRecWithCurrency>(writer).WithFirstLineHeader().QuoteAllFields())
    {
        parser.Write(objs);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

20.3 Enum Support

Cinchoo ETL implicitly handles parsing/writing of enum column values from CSV files. If you want to fine control the parsing of these values, you can specify them globally via ChoTypeConverterFormatSpec.EnumFormat. Default is ChoEnumFormatSpec.Value

FYI, changing this value will impact system wide.

There are 3 possible values can be used

  1. ChoEnumFormatSpec.Value – Enum value is used for parsing.
  2. ChoEnumFormatSpec.Name – Enum key name is used for parsing.
  3. ChoEnumFormatSpec.Description – If each enum key is decorated with DescriptionAttribute, its value will be use for parsing.

Listing 20.3.1 Specifying Enum format specs during parsing

public enum EmployeeType
{
    [Description("Full Time Employee")]
    Permanent = 0,
    [Description("Temporary Employee")]
    Temporary = 1,
    [Description("Contract Employee")]
    Contract = 2
}
 
static void EnumTest()
{
    ChoTypeConverterFormatSpec.Instance.EnumFormat = ChoEnumFormatSpec.Description;
 
    List<ExpandoObject> objs = new List<ExpandoObject>();
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 10;
    rec1.Name = "Mark";
    rec1.JoinedDate = new DateTime(2001, 2, 2);
    rec1.IsActive = true;
    rec1.Salary = new ChoCurrency(100000);
    rec1.Status = EmployeeType.Permanent;
    objs.Add(rec1);
 
    dynamic rec2 = new ExpandoObject();
    rec2.Id = 200;
    rec2.Name = "Lou";
    rec2.JoinedDate = new DateTime(1990, 10, 23);
    rec2.IsActive = false;
    rec2.Salary = new ChoCurrency(150000);
    rec2.Status = EmployeeType.Contract;
    objs.Add(rec2);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer).WithFirstLineHeader().QuoteAllFields())
    {
        parser.Write(objs);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

20.4 Boolean Support

Cinchoo ETL implicitly handles parsing/writing of boolean CSV column values from CSV files. If you want to fine control the parsing of these values, you can specify them globally via ChoTypeConverterFormatSpec.BooleanFormat. Default value is ChoBooleanFormatSpec.ZeroOrOne

FYI, changing this value will impact system wide.

There are 4 possible values can be used

  1. ChoBooleanFormatSpec.ZeroOrOne – ‘0’ for false. ‘1’ for true.
  2. ChoBooleanFormatSpec.YOrN – ‘Y’ for true, ‘N’ for false.
  3. ChoBooleanFormatSpec.TrueOrFalse – ‘True’ for true, ‘False’ for false.
  4. ChoBooleanFormatSpec.YesOrNo – ‘Yes’ for true, ‘No’ for false.

Listing 20.4.1 Specifying boolean format specs during parsing

static void BoolTest()
{
    ChoTypeConverterFormatSpec.Instance.BooleanFormat = ChoBooleanFormatSpec.YOrN;
 
    List<ExpandoObject> objs = new List<ExpandoObject>();
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 10;
    rec1.Name = "Mark";
    rec1.JoinedDate = new DateTime(2001, 2, 2);
    rec1.IsActive = true;
    rec1.Salary = new ChoCurrency(100000);
    rec1.Status = EmployeeType.Permanent;
    objs.Add(rec1);
 
    dynamic rec2 = new ExpandoObject();
    rec2.Id = 200;
    rec2.Name = "Lou";
    rec2.JoinedDate = new DateTime(1990, 10, 23);
    rec2.IsActive = false;
    rec2.Salary = new ChoCurrency(150000);
    rec2.Status = EmployeeType.Contract;
    objs.Add(rec2);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer).WithFirstLineHeader().QuoteAllFields())
    {
        parser.Write(objs);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

20.5 DateTime Support

Cinchoo ETL implicitly handles parsing/writing of datetime CSV column values from CSV files using system Culture or custom set culture. If you want to fine control the parsing of these values, you can specify them globally via ChoTypeConverterFormatSpec.DateTimeFormat. Default value is ‘d’.

FYI, changing this value will impact system wide.

You can use any valid standard or custom datetime .NET format specification to parse the datetime CSV values from the file.

Listing 20.5.1 Specifying datetime format specs during parsing

static void DateTimeDynamicTest()
{
    ChoTypeConverterFormatSpec.Instance.DateTimeFormat = "MMM dd, yyyy";
 
    List<ExpandoObject> objs = new List<ExpandoObject>();
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 10;
    rec1.Name = "Mark";
    rec1.JoinedDate = new DateTime(2001, 2, 2);
    rec1.IsActive = true;
    rec1.Salary = new ChoCurrency(100000);
    objs.Add(rec1);
 
    dynamic rec2 = new ExpandoObject();
    rec2.Id = 200;
    rec2.Name = "Lou";
    rec2.JoinedDate = new DateTime(1990, 10, 23);
    rec2.IsActive = false;
    rec2.Salary = new ChoCurrency(150000);
    objs.Add(rec2);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer).WithFirstLineHeader().QuoteAllFields())
    {
        parser.Write(objs);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

Sample above shows how to generate custom datetime values to CSV file.

Note: As the datetime values contains CSV separator, we instruct the writer to quote all fields.

21. Fluent API

CSVWriter exposes few frequent to use configuration parameters via fluent API methods. This will make the programming of generating CSV files quicker.

21.1 WithDelimiter

This API method sets the CSV field separator on CSVWriter.

static void QuickDynamicDelimiterTest()
{
    List<ExpandoObject> objs = new List<ExpandoObject>();
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 10;
    rec1.Name = "Mark";
    rec1.JoinedDate = new DateTime(2001, 2, 2);
    rec1.IsActive = true;
    rec1.Salary = new ChoCurrency(100000);
    objs.Add(rec1);
 
    dynamic rec2 = new ExpandoObject();
    rec2.Id = 200;
    rec2.Name = "Lou";
    rec2.JoinedDate = new DateTime(1990, 10, 23);
    rec2.IsActive = false;
    rec2.Salary = new ChoCurrency(150000);
    objs.Add(rec2);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer).WithDelimiter("|"))
    {
        parser.Write(objs);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

21.2 WithFirstLineHeader

This API method flags the CSV file contains first row as header or not. Optional bool parameter specifies the first row header or not. Default is true.

static void QuickDynamicTest()
{
    List<ExpandoObject> objs = new List<ExpandoObject>();
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 10;
    rec1.Name = "Mark";
    rec1.JoinedDate = new DateTime(2001, 2, 2);
    rec1.IsActive = true;
    rec1.Salary = new ChoCurrency(100000);
    objs.Add(rec1);
 
    dynamic rec2 = new ExpandoObject();
    rec2.Id = 200;
    rec2.Name = "Lou";
    rec2.JoinedDate = new DateTime(1990, 10, 23);
    rec2.IsActive = false;
    rec2.Salary = new ChoCurrency(150000);
    objs.Add(rec2);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer).WithDelimiter("|").WithFirstLineHeader())
    {
        parser.Write(objs);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

21.3 WithFields

This API method specifies the list of CSV fields to be considered for writing CSV file. Other fields will be discarded. Field names are case-insensitive.

static void QuickDynamicTest()
{
    List<ExpandoObject> objs = new List<ExpandoObject>();
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 10;
    rec1.Name = "Mark";
    rec1.JoinedDate = new DateTime(2001, 2, 2);
    rec1.IsActive = true;
    rec1.Salary = new ChoCurrency(100000);
    objs.Add(rec1);
 
    dynamic rec2 = new ExpandoObject();
    rec2.Id = 200;
    rec2.Name = "Lou";
    rec2.JoinedDate = new DateTime(1990, 10, 23);
    rec2.IsActive = false;
    rec2.Salary = new ChoCurrency(150000);
    objs.Add(rec2);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer).WithFirstLineHeader().WithFields("Id", "Name"))
    {
        parser.Write(objs);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

21.4 WithField

This API method used to add CSV column with specific date type, quote flag, and/or quote character. This method helpful in dynamic object model, by specifying each and individual CSV column with appropriate datatype.

static void QuickDynamicTest()
{
    List<ExpandoObject> objs = new List<ExpandoObject>();
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 10;
    rec1.Name = "Mark";
    rec1.JoinedDate = new DateTime(2001, 2, 2);
    rec1.IsActive = true;
    rec1.Salary = new ChoCurrency(100000);
    objs.Add(rec1);
 
    dynamic rec2 = new ExpandoObject();
    rec2.Id = 200;
    rec2.Name = "Lou";
    rec2.JoinedDate = new DateTime(1990, 10, 23);
    rec2.IsActive = false;
    rec2.Salary = new ChoCurrency(150000);
    objs.Add(rec2);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer).WithFirstLineHeader().WithField("Id", typeof(int)).WithField("Name"))
    {
        parser.Write(objs);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

21.5 QuoteAllFields

This API method used to specify whether all fields are to be surrounded by quotes or not.

static void QuickDynamicTest()
{
    List<ExpandoObject> objs = new List<ExpandoObject>();
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 10;
    rec1.Name = "Mark";
    rec1.JoinedDate = new DateTime(2001, 2, 2);
    rec1.IsActive = true;
    rec1.Salary = new ChoCurrency(100000);
    objs.Add(rec1);
 
    dynamic rec2 = new ExpandoObject();
    rec2.Id = 200;
    rec2.Name = "Lou";
    rec2.JoinedDate = new DateTime(1990, 10, 23);
    rec2.IsActive = false;
    rec2.Salary = new ChoCurrency(150000);
    objs.Add(rec2);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer).WithFirstLineHeader().QuoteAllFields())
    {
        parser.Write(objs);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

21.6 ColumnCountStrict

This API method used to set the CSVWriter to perform check on column countnness before writing CSV file.

static void QuickDynamicTest()
{
    List<ExpandoObject> objs = new List<ExpandoObject>();
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 10;
    rec1.Name = "Mark";
    rec1.JoinedDate = new DateTime(2001, 2, 2);
    rec1.IsActive = true;
    rec1.Salary = new ChoCurrency(100000);
    objs.Add(rec1);
 
    dynamic rec2 = new ExpandoObject();
    rec2.Id = 200;
    rec2.Name = "Lou";
    rec2.JoinedDate = new DateTime(1990, 10, 23);
    rec2.IsActive = false;
    rec2.Salary = new ChoCurrency(150000);
    objs.Add(rec2);
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVWriter(writer).WithFirstLineHeader().ColumnCountStrict())
    {
        parser.Write(objs);
 
        writer.Flush();
        stream.Position = 0;
 
        Console.WriteLine(reader.ReadToEnd());
    }
}

Cinchoo ETL – CSV Reader

Download source code

Download binary

Contents

1. Introduction

ChoETL is an open source ETL (extract, transform and load) framework for .NET. It is a code based library for extracting data from multiple sources, transforming, and loading into your very own data warehouse in .NET environment. You can have data in your data warehouse in no time.

This article talks about using CSVReader component offered by ChoETL framework. It is a simple utility class to extract CSV data from file / source.

UPDATE: Corresponding CSVWriter article can be found here.

Features:

  • Follows CSV standard file rules. Gracefully handles data fields that contain commas and line breaks.
  • In addition to comma, most delimiting characters can be used, including tab delimited fields.
  • Exposes IEnumarable list of objects – which is often used with LINQ query for projection, aggregation and filtration etc.
  • Supports deferred reading.
  • Supports processing files with culture specific date, currency and number formats.
  • Supports different character encoding.
  • Recognizes a wide variety of date, currency, enum, boolean and number formats when reading files.
  • Provides fine control of date, currency, enum, boolean, number formats when writing files.
  • Detailed and robust error handling, allowing you to quickly find and fix problems.

2. Requirement

This framework library is written in C# using .NET 4.5 Framework.

3. “Hello World!” Sample

  • Open VS.NET 2013 or higher
  • Create a sample VS.NET (.NET Framework 4.5) Console Application project
  • Install ChoETL via Package Manager Console using Nuget Command: Install-Package ChoETL
  • Use the ChoETL namespace

Let’s begin by looking into a simple example of reading CSV file having 2 columns

Listing 3.1 Sample CSV data file

1,Tom
2,Carl
3,Mark

There are number of ways you can get the CSV file parsing started with minimal setup

3.1. Quick load – Data First Approach

It is the quick way to load a CSV file in no time. No POCO object is required. Sample code below shows how to load the file

Listing 3.1.1 Load CSV file using iterator

foreach (var e in new ChoCSVReader("Emp.csv"))
    Console.WriteLine(e.ToStringEx());

Listing 3.1.2 Load CSV file using loop

var reader = new ChoCSVReader("Emp.csv");
var rec = (object)null;
 
while ((rec = reader.Read()) != null)
    Console.WriteLine(rec.ToStringEx());

This model uses generaric CSVReader object for parsing the file. CSVReader auto name the columns as Column1, Column2 … in the dynamic object.

3.2. Code First Approach

This is another way to parse and load CSV file using POCO class. First define a simple data class to match the underlying CSV file layout

Listing 3.2.1 Simple POCO entity class

public partial class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; } 
}

In above, the class defines two properties matching the sample CSV file template.

Listing 3.2.2 Load CSV file

foreach (var e in new ChoCSVReader<EmployeeRec>("Emp.csv"))
    Console.WriteLine(e.ToStringEx());

3.3. Configuration First Approach

In this model, we define the CSV configuration with all the necessary parsing parameters along with CSV columns matching with the underlying CSV file.

Listing 3.3.1 Define CSV configuration

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1));
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));

In above, the class defines two properties matching the sample CSV file template.

Listing 3.3.2 Load CSV file without POCO object

foreach (var e in new ChoCSVReader("Emp.csv", config))
    Console.WriteLine(e.ToStringEx());

Listing 3.3.3 Load CSV file with POCO object

foreach (var e in new ChoCSVReader<EmployeeRec>("Emp.csv", config))
    Console.WriteLine(e.ToStringEx());

3.4. Code First with declarative configuration

This is the combined approach to define POCO entity class along with CSV configuration parameters decorated declaratively. id is required column and name is optional value column with default value XXXX“. If name is not present, it will take the default value.

Listing 3.4.1 Define POCO Object

public class EmployeeRec
{
    [ChoCSVRecordField(1)]
    [Required]
    public int Id
    {
        get;
        set;
    }
    [ChoCSVRecordField(2)]
    [DefaultValue("XXXX")]
    public string Name
    {
        get;
        set;
    }

    public override string ToString()
    {
        return "{0}. {1}.".FormatString(Id, Name);
    }
}

The code above illustrates about defining POCO object to carry the values of each record line in the input file. First thing defines property for each record field with ChoCSVRecordFieldAttribute to qualify for CSV record mapping. Each property must specify position in order to be mapped to CSV column. Position is 1 based. Id is a required property. We decorated it with RequiredAttribute. Name is given default value using DefaultValueAttribute. It means that if the Name CSV column contains empty value in the file, it will be defaulted to ‘XXXX‘ value.

It is very simple and ready to extract CSV data in no time.

Listing 3.4.2 Main Method

class Program
{
    static void Main(string[] args)
    {
        using (var stream = new MemoryStream())
        using (var reader = new StreamReader(stream))
        using (var writer = new StreamWriter(stream))
        using (var parser = new ChoCSVReader<EmployeeRec>(reader))
        {
            writer.WriteLine("1,Carl");
            writer.WriteLine("2,Mark");
            writer.WriteLine("3,");

            writer.Flush();
            stream.Position = 0;
 
            object row = null;
 
            while ((row = parser.Read()) != null)
                Console.WriteLine(row.ToString());
        }
    }
}

We start by creating a new instance of ChoCSVReader object. That’s all. All the heavy lifting of parsing and loading CSV data stream into the objects is done by the parser under the hood.

By default, CSVReader discovers and uses default configuration parameters while loading CSV file. These can be overridable according to your needs. The following sections will give details about each configuration attributes.

4. Reading All Records

It is as easy as setting up POCO object match up with CSV file structure, you can read the whole file as enumerable pattern. It is a deferred execution mode, but take care while making any aggregate operation on them. This will load the entire file records into memory.

Listing 4.1 Read CSV File

foreach (var e in new ChoCSVReader<EmployeeRec>("Emp.csv"))
    Console.WriteLine(e.ToString());

or:

Listing 4.2 Read CSV file stream

foreach (var e in new ChoCSVReader<EmployeeRec>(textReader))
    Console.WriteLine(e.ToString());

This model keeps your code elegant, clean, easy to read and maintain. Also leverages LINQ extension methods to to perform grouping, joining, projection, aggregation etc.

Listing 4.3 Using LINQ

var list = (from o in new ChoCSVReader<EmployeeRec>("Emp.csv")
           where o.Name != null && o.Name.StartsWith("R")
           select o).ToArray();
 
foreach (var e in list)
    Console.WriteLine(e.ToStringEx());

5. Read Records Manually

It is as easy as setting up POCO object match up with CSV file structure, you can read the whole file as enumerable pattern

Listing 5.1 Read CSV file

var reader = new ChoCSVReader<EmployeeRec>("Emp.csv");
var rec = (object)null;
 
while ((rec = reader.Read()) != null)
    Console.WriteLine(rec.ToStringEx());

6. Customize CSV Record

Using ChoCSVRecordObjectAttribute, you can customize the POCO entity object declaratively.

Listing 6.1 Customizing POCO object for each record

[ChoCSVRecordObject(Encoding = "Encoding.UTF32", 
ErrorMode = ChoErrorMode.IgnoreAndContinue, IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All)]
public class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "id")]
    public int Id { get; set; }
    [ChoCSVRecordField(2, FieldName ="Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available attributes to carry out customization of CSV load operation on a file.

  • Delimiter – The value used to separate the fields in a CSV row. Default is Culture.TextInfo.ListSeparator used.
  • EOLDelimiter – The value used to separate CSV rows. Default is \r\n (NewLine).
  • CultureName – The culture name (ex. en-US, en-GB) used to read and write CSV data.
  • IgnoreEmptyLine – A flag to let the reader know if a record should be skipped when reading if it’s empty. A record is considered empty if all fields are empty.
  • Comments – The value used to denote a line that is commented out. Multiple comments can be specified. Must be separated by comma.
  • QuoteChar – The value used to escape fields that contain a delimiter, quote, or line ending.
  • QuoteAllFields – N/A for reader.
  • Encoding – The encoding of the CSV file.
  • HasExcelSeperator – N/A for reader. Reader seamlessly recognize the excel separator if specified in the CSV file and use them for parsing.
  • ColumnCountStrict – This flag indicates if an exception should be thrown if reading an expected field is missing.
  • ColumnOrderStrict – This flag indicates if an exception should be thrown if reading an expected field is in wrong position in the file. This check will be performed only when ColumnCountStrict is true.
  • BufferSize – The size of the internal buffer that is used when reader is from the StreamReader.
  • ErrorMode – This flag indicates if an exception should be thrown if reading and an expected field is failed to load. This can be overridden per property. Possible values are:
    • IgnoreAndContinue – Ignore the error, record will be skipped and continue with next.
    • ReportAndContinue – Report the error to POCO entity if it is of IChoNotifyRecordRead type
    • ThrowAndStop – Throw the error and stop the execution
  • IgnoreFieldValueMode – A flag to let the reader know if a record should be skipped when reading if it’s empty / null. This can be overridden per property. Possible values are:
    • Null – N/A
    • DBNull – N/A
    • Empty – skipped if the record value is empty
    • WhiteSpace – skipped if the record value contains only whitespaces
  • ObjectValidationMode – A flag to let the reader know about the type of validation to be performed with record object. Possible values are:
    • Off – No object validation performed.
    • MemberLevel – Validation performed at the time of each CSV property gets loaded with value.
    • ObjectLevel – Validation performed after all the properties are loaded to the POCO object.

7. Customize CSV Header

If the CSV file has header, you can instruct the POCO entity by using ChoCSVFileHeaderAttribute.

Listing 6.1 Customizing POCO object for file header

[ChoCSVFileHeader]
public class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "id")]
    public int Id { get; set; }
    [ChoCSVRecordField(2, FieldName ="Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available members to add some customization to it according to your need.

  • FillChar – N/A for reader
  • Justification – N/A for reader
  • TrimOption – This flag tells the reader to trim whitespace from the beginning and ending of the CSV column header when reading. Possible values are Trim, TrimStart, TrimEnd.
  • Truncate – N/A for reader

8. Customize CSV Fields

For each CSV column, you can specify the mapping in POCO entity property using ChoCSVRecordFieldAttribute.

Listing 6.1 Customizing POCO object for CSV columns

[ChoCSVFileHeader]
public class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "id")]
    public int Id { get; set; }
    [ChoCSVRecordField(2, FieldName ="Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available members to add some customization to it for each property:

  • FieldPosition – When mapping by position, you specify the index of the CSV column that you want to use for that property. It is 1 based.
  • FieldName – When mapping by name, you specify the name of the CSV column that you want to use for that property. For this to work, the CSV file must have a header record. The name you specify must match with the name of the header record.
  • FillChar – N/A for reader.
  • FieldValueJustification – N/A for reader.
  • FieldValueTrimOption – This flag tells the reader to trim whitespace from the beginning and ending of the field value when reading. Possible values are Trim, TrimStart, TrimEnd.
  • Truncate – N/A for reader.
  • Size – N/A for reader.
  • QuoteField – A flag that tells the reader that the CSV column value is surrounded by quotes.
  • ErrorMode – This flag indicates if an exception should be thrown if reading and an expected field failed to load. Possible values are:
    • IgnoreAndContinue – Ignore the error and continue to load other properties of the record.
    • ReportAndContinue – Report the error to POCO entity if it is of IChoRecord type.
    • ThrowAndStop – Throw the error and stop the execution.
  • IgnoreFieldValueMode – A flag to let the reader know if a record should be skipped when reading if it’s empty / null. Possible values are:
    • Null – N/A
    • DBNull – N/A
    • Empty – skipped if the record value is empty.
    • WhiteSpace – skipped if the record value contains only whitespaces.

8.1. DefaultValue

It is the value used and set to the property when the CSV value is empty or whitespace (controlled via IgnoreFieldValueMode).

Any POCO entity property can be specified with default value using System.ComponentModel.DefaultValueAttribute.

8.2. ChoFallbackValue

It is the value used and set to the property when the CSV value failed to set. Fallback value only set when ErrorMode is either IgnoreAndContinue or ReportAndContinue.

Any POCO entity property can be specified with fallback value using ChoETL.ChoFallbackValueAttribute.

8.3. Type Converters

Most of the primitive types are automatically converted and set them to the properties. If the value of the CSV field can’t automatically be converted into the type of the property, you can specify a custom / built-in .NET converters to convert the value. These can be either IValueConverter or TypeConverter converters.

There are couple of ways you can specify the converters for each field

  • Declarative Approach
  • Configuration Approach

8.3.1. Declarative Approach

This model is applicable to POCO entity object only. If you have POCO class, you can specify the converters to each property to carry out necessary conversion on them. Samples below shows the way to do it.

Listing 8.3.1.1 Specifying type converters

[ChoCSVFileHeader]
public class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    public int Id { get; set; }
    [ChoCSVRecordField(2, FieldName ="Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Listing 8.3.1.2 IntConverter implementation

public class IntConverter : IValueConverter
{
    public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
    {
        return value;
    }
 
    public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
    {
        return value;
    }
}

In the example above, we defined custom IntConverter class. And showed how to use it with ‘IdCSV property.

8.3.2. Configuration Approach

This model is applicable to both dynamic and POCO entity object. This gives freedom to attach the converters to each property at runtime. This takes the precedence over the declarative converters on POCO classes.

Listing 8.3.2.2 Specifying TypeConverters

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.FileHeaderConfiguration.HasHeaderRecord = true;
config.ThrowAndStopOnMissingField = false;

ChoCSVRecordFieldConfiguration idConfig = new ChoCSVRecordFieldConfiguration("Id", 1);
idConfig.AddConverter(new IntConverter());
config.RecordFieldConfigurations.Add(idConfig);

config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name1", 2));

In above, we construct and attach the IntConverter to ‘Id’ field using AddConverter helper method in ChoCSVRecordFieldConfiguration object.

Likewise, if you want to remove any converter from it, you can use RemoveConverter on ChoCSVRecordFieldConfiguration object.

8.4. Validations

CSVReader leverages both System.ComponentModel.DataAnnotations and Validation Block validation attributes to specify validation rules for individual fields of POCO entity. Refer to the MSDN site for a list of available DataAnnotations validation attributes.

Listing 8.4.1 Using validation attributes in POCO entity

[ChoCSVFileHeader]
[ChoCSVRecordObject(Encoding = "Encoding.UTF32", ErrorMode = ChoErrorMode.IgnoreAndContinue,
       IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, ThrowAndStopOnMissingField = false)]
public partial class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
 
    [ChoCSVRecordField(2, FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
}

In example above, used Range validation attribute for Id property. Required validation attribute to Name property. CSVReader performs validation on them during load based on Configuration.ObjectValidationMode is set to ChoObjectValidationMode.MemberLevel or ChoObjectValidationMode.ObjectLevel.

Sometime you may want override the defined declarative validation behaviors comes with POCO class, you can do with Cinchoo ETL via configuration approach. The sample below shows the way to override them.

static void ValidationOverridePOCOTest()
{
    ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
    var idConfig = new ChoCSVRecordFieldConfiguration("Id", 1);
    idConfig.Validators = new ValidationAttribute[] { new RequiredAttribute() };
    config.RecordFieldConfigurations.Add(idConfig);
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Salary", 3) { FieldType = typeof(ChoCurrency) });
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader<EmployeeRecWithCurrency>(reader, config))
    {
        writer.WriteLine(",Carl,$100000");
        writer.WriteLine("2,Mark,$50000");
        writer.WriteLine("3,Tom,1000");
 
        writer.Flush();
        stream.Position = 0;
 
        object rec;
        while ((rec = parser.Read()) != null)
        {
            Console.WriteLine(rec.ToStringEx());
        }
    }
}
 
public class EmployeeRecWithCurrency
{
    public int? Id { get; set; }
    public string Name { get; set; }
    public ChoCurrency Salary { get; set; }
}

Some cases, you may want to take control and perform manual self validation within the POCO entity class. This can be achieved by inheriting POCO object from IChoValidatable interface.

Listing 8.4.2 Manual validation on POCO entity

[ChoCSVFileHeader]
[ChoCSVRecordObject(Encoding = "Encoding.UTF32", ErrorMode = ChoErrorMode.IgnoreAndContinue,
       IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, ThrowAndStopOnMissingField = false)]
public partial class EmployeeRec : IChoValidatable
{
    [ChoCSVRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
 
    [ChoCSVRecordField(2, FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool TryValidate(object target, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public bool TryValidateFor(object target, string memberName, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public void Validate(object target)
    {
    }
 
    public void ValidateFor(object target, string memberName)
    {
    }
}

Sample above shows how to implement custom self-validation in POCO object.

IChoValidatable interface exposes below methods

  • TryValidate – Validate entire object, return true if all validation passed. Otherwise return false.
  • Validate – Validate entire object, throw exception if validation is not passed.
  • TryValidateFor – Validate specific property of the object, return true if all validation passed. Otherwise return false.
  • ValidateFor – Validate specific property of the object, throw exception if validation is not passed.

9. Excel Field Separator

If the input CSV file contains Excel field separator, the reader automatically detects and use them as field separator.

Listing 9.1 Sample CSV file with Excel field separator

sep=,
1,"Eldon Base for stackable storage shelf, platinum"
2,"1.7 Cubic Foot Compact ""Cube"" Office Refrigerators"
3,"Cardinal Slant-D® Ring Binder, Heavy Gauge Vinyl"
4,R380
5,Holmes HEPA Air Purifier

10. Callback Mechanism

CSVReader offers industry standard CSV parsing out of the box to handle most of the parsing needs. If the parsing is not handling any of the needs, you can use the callback mechanism offered by CSVReader to handle such situations. In order to participate in the callback mechanism, Either POCO entity object or DataAnnotation’s MetadataType type object must be inherited by IChoNotifyRecordRead interface.

Tip: Any exceptions raised out of these interface methods will be ignored.

IChoNotifyRecordRead exposes the below methods:

  • BeginLoad – Invoked at the begin of the CSV file load
  • EndLoad – Invoked at the end of the CSV file load
  • BeforeRecordLoad – Raised before the CSV record load
  • AfterRecordLoad – Raised after CSV record load
  • RecordLoadError – Raised when CSV record load errors out
  • BeforeRecordFieldLoad – Raised before CSV column value load
  • AfterRecordFieldLoad – Raised after CSV column value load
  • RecordFieldLoadError – Raised when CSV column value errors out

Listing 10.1 Direct POCO callback mechanism implementation

[ChoCSVFileHeader]
[ChoCSVRecordObject(Encoding = "Encoding.UTF32", ErrorMode = ChoErrorMode.IgnoreAndContinue,
       IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, ThrowAndStopOnMissingField = false)]
public partial class EmployeeRec : IChoNotifyRecordRead
{
    [ChoCSVRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
    
    [ChoCSVRecordField(2, FieldName = "Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordFieldLoad(object target, int index, string propName, object value)
    {
        throw new NotImplementedException();
    }
 
    public bool AfterRecordLoad(object target, int index, object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordFieldLoad(object target, int index, string propName, ref object value)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordLoad(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeginLoad(object source)
    {
        throw new NotImplementedException();
    }
 
    public void EndLoad(object source)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordFieldLoadError(object target, int index, string propName, object value, Exception ex)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordLoadError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
}

Listing 10.2 MetaDataType based callback mechanism implementation

[ChoCSVFileHeader]
[ChoCSVRecordObject(Encoding = "Encoding.UTF32", ErrorMode = ChoErrorMode.IgnoreAndContinue,
       IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, ThrowAndStopOnMissingField = false)]
public class EmployeeRecMeta : IChoNotifyRecordRead
{
    [ChoCSVRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }

    [ChoCSVRecordField(2, FieldName = "Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordFieldLoad(object target, int index, string propName, object value)
    {
        throw new NotImplementedException();
    }
 
    public bool AfterRecordLoad(object target, int index, object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordFieldLoad(object target, int index, string propName, ref object value)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordLoad(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeginLoad(object source)
    {
        throw new NotImplementedException();
    }
 
    public void EndLoad(object source)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordFieldLoadError(object target, int index, string propName, object value, Exception ex)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordLoadError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
} 

[MetadataType(typeof(EmployeeRecMeta))]
public partial class EmployeeRec
{
    [ChoCSVRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
    
    [ChoCSVRecordField(2, FieldName = "Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
}

10.1 BeginLoad

This callback invoked once at the beginning of the CSV file load. source is the CSV file stream object. In here you have chance to inspect the stream, return true to continue the CSV load. Return false to stop the parsing.

Listing 10.1.1 BeginLoad Callback Sample

public bool BeginLoad(object source)
{
    StreamReader sr = source as StreamReader;
    return true;
}

10.2 EndLoad

This callback invoked once at the end of the CSV file load. source is the CSV file stream object. In here you have chance to inspect the stream, do any post steps to be performed on the stream.

Listing 10.2.1 EndLoad Callback Sample

public void EndLoad(object source)
{
    StreamReader sr = source as StreamReader;
}

10.3 BeforeRecordLoad

This callback invoked before each record line in the CSV file is loaded. target is the instance of the POCO record object. index is the line index in the file. source is the CSV record line. In here you have chance to inspect the line, and override it with new line if want to.

TIP: If you want to skip the line from loading, set the source to null.

TIP: If you want to take control of parsing and loading the record properties by yourself, set the source to String.Empty. 

Return true to continue the load process, otherwise return false to stop the process.

Listing 10.3.1 BeforeRecordLoad Callback Sample

public bool BeforeRecordLoad(object target, int index, ref object source)
{
    string line = source as string;
    return true;
}

10.4 AfterRecordLoad

This callback invoked after each record line in the CSV file is loaded. target is the instance of the POCO record object. index is the line index in the file. source is the CSV record line. In here you have chance to do any post step operation with the record line.

Return true to continue the load process, otherwise return false to stop the process.

Listing 10.4.1 AfterRecordLoad Callback Sample

public bool AfterRecordLoad(object target, int index, object source)
{
    string line = source as string;
    return true;
}

10.5 RecordLoadError

This callback invoked if error encountered while loading record line. target is the instance of the POCO record object. index is the line index in the file. source is the CSV record line. ex is the exception object. In here you have chance to handle the exception. This method invoked only when Configuration.ErrorMode is ReportAndContinue.

Return true to continue the load process, otherwise return false to stop the process.

Listing 10.5.1 RecordLoadError Callback Sample

public bool RecordLoadError(object target, int index, object source, Exception ex)
{
    string line = source as string;
    return true;
}

10.6 BeforeRecordFieldLoad

This callback invoked before each CSV record column is loaded. target is the instance of the POCO record object. index is the line index in the file. propName is the CSV record property name. value is the CSV column value. In here you have chance to inspect the CSV record property value and perform any custom validations etc.

Return true to continue the load process, otherwise return false to stop the process.

Listing 10.6.1 BeforeRecordFieldLoad Callback Sample

public bool BeforeRecordFieldLoad(object target, int index, string propName, ref object value)
{
    return true;
}

10.7 AfterRecordFieldLoad

This callback invoked after each CSV record column is loaded. target is the instance of the POCO record object. index is the line index in the file. propName is the CSV record property name. value is the CSV column value. Any post field operation can be performed here, like computing other properties, validations etc.

Return true to continue the load process, otherwise return false to stop the process.

Listing 10.7.1 AfterRecordFieldLoad Callback Sample

public bool AfterRecordFieldLoad(object target, int index, string propName, object value)
{
    return true;
}

10.8 RecordLoadFieldError

This callback invoked when error encountered while loading CSV record column value. target is the instance of the POCO record object. index is the line index in the file. propName is the CSV record property name. value is the CSV column value. ex is the exception object. In here you have chance to handle the exception. This method invoked only after the below two sequences of steps performed by the CSVReader

  • CSVReader looks for FallbackValue value of each CSV property. If present, it tries to assign its value to it.
  • If the FallbackValue value not present and the Configuration.ErrorMode is specified as ReportAndContinue., this callback will be executed.

Return true to continue the load process, otherwise return false to stop the process.

Listing 10.8.1 RecordFieldLoadError Callback Sample

public bool RecordFieldLoadError(object target, int index, string propName, object value, Exception ex)
{
    return true;
}

10. Customization

CSVReader automatically detects and loads the configured settings from POCO entity. At runtime, you can customize and tweak these parameters before CSV parsing. CSVReader exposes Configuration property, it is of ChoCSVRecordConfiguration object. Using this property, you can customize them.

Listing 10.1 Customizing CSVReader at run-time

class Program
{
    static void Main(string[] args)
    {
        using (var stream = new MemoryStream())
        using (var reader = new StreamReader(stream))
        using (var writer = new StreamWriter(stream))
        using (var parser = new ChoCSVReader<EmployeeRec>(reader))
        {
            writer.WriteLine("1,Carl");
            writer.WriteLine("2,Mark");
            writer.WriteLine("3,");

            writer.Flush();
            stream.Position = 0;
 
            object row = null;
  
            parser.Configuration.ColumnCountStrict = true;
            while ((row = parser.Read()) != null)
                Console.WriteLine(row.ToString());
        }
    }

11. AsDataReader Helper Method

CSVReader exposes AsDataReader helper method to retrieve the CSV records in .NET datareader object. DataReader are fast-forward streams of data. This datareader can be used in few places like bulk coping data to database using SqlBulkCopy, loading disconnected DataTable, etc.

Listing 11.1 Reading as DataReader sample

static void AsDataReaderTest()
{
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader<EmployeeRec>(reader))
    {
        writer.WriteLine("1,Carl");
        writer.WriteLine("2,Mark");
        writer.WriteLine("3,Tom");
 
        writer.Flush();
        stream.Position = 0;
 
        IDataReader dr = parser.AsDataReader();
        while (dr.Read())
        {
            Console.WriteLine("Id: {0}, Name: {1}", dr[0], dr[1]);
        }
    }
}

12. AsDataTable Helper Method

CSVReader exposes AsDataTable helper method to retrieve the CSV records in .NET DataTable object. It then can be persisted to disk, displayed in grid/controls or stored in memory like any other object.

Listing 12.1 Reading as DataTable sample

static void AsDataTableTest()
{
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader<EmployeeRec>(reader))
    {
        writer.WriteLine("1,Carl");
        writer.WriteLine("2,Mark");
        writer.WriteLine("3,Tom");
 
        writer.Flush();
        stream.Position = 0;
 
        DataTable dt = parser.AsDataTable();
        foreach (DataRow dr in dt.Rows)
        {
            Console.WriteLine("Id: {0}, Name: {1}", dr[0], dr[1]);
        }
    }
}

13. Using Dynamic Object

So far, the article explained about using CSVReader with POCO object. CSVReader also supports loading CSV file without POCO object. It leverages .NET dynamic feature. The sample below shows how to read CSV stream without POCO object.

If you have CSV file, you can parse and load the file with minimal/zero configuration. If the CSV file does not have header record line, the parser automatically names the columns as Column1, Column2, etc.

The sample below shows it:

Listing 13.1 Loading CSV file without header sample

class Program
{
    static void Main(string[] args)
    {
        dynamic row;
        using (var stream = new MemoryStream())
        using (var reader = new StreamReader(stream))
        using (var writer = new StreamWriter(stream))
        using (var parser = new ChoCSVReader(reader))
        {
            writer.WriteLine("1,Carl");
            writer.WriteLine("2,Mark");
            writer.Flush();
            stream.Position = 0;
 
            while ((row = parser.Read()) != null)
            {
                Console.WriteLine(row.Column1);
            }
        }
    }
}

If the CSV file has a header, you can state that in the configuration as HasHeaderRecord is true and parse the file as simple as below:

Listing 13.2 Loading CSV file with header sample

class Program
{
    static void Main(string[] args)
    {
        ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
        config.CSVFileHeaderConfiguration.HasHeaderRecord = true;

        dynamic row;
        using (var stream = new MemoryStream())
        using (var reader = new StreamReader(stream))
        using (var writer = new StreamWriter(stream))
        using (var parser = new ChoCSVReader(reader. config))
        {
            writer.WriteLine("Id,Name");
            writer.WriteLine("1,Carl");
            writer.WriteLine("2,Mark");
            writer.Flush();
            stream.Position = 0;
 
            while ((row = parser.Read()) != null)
            {
                Console.WriteLine(row.Name);
            }
        }
    }
}

The above example automatically discovers the CSV columns from the header and parses the file.

You can override the default behavior of discovering columns automatically by adding field configurations manually and pass it to CSVReader for parsing file.

Sample shows how to do it:

Listing 13.3 Loading CSV file with configuration

class Program
{
    static void Main(string[] args)
    {
        ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
        config.CSVFileHeaderConfiguration.HasHeaderRecord = true;
        config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1));
        config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));

        dynamic row;
        using (var stream = new MemoryStream())
        using (var reader = new StreamReader(stream))
        using (var writer = new StreamWriter(stream))
        using (var parser = new ChoCSVReader(reader. config))
        {
            writer.WriteLine("Id,Name");
            writer.WriteLine("1,Carl");
            writer.WriteLine("2,Mark");
            writer.Flush();
            stream.Position = 0;
 
            while ((row = parser.Read()) != null)
            {
                Console.WriteLine(row.Name);
            }
        }
    }
}

To completely turn off the auto column discovery, you will have to set ChoCSVRecordConfiguration.AutoDiscoverColumns to false.

13.1. DefaultValue

It is the value used and set to the property when the CSV value is empty or whitespace (controlled via IgnoreFieldValueMode).

Any POCO entity property can be specified with default value using System.ComponentModel.DefaultValueAttribute.

For dynamic object members or to override the declarative POCO object member’s default value specification, you can do so through configuration as shown below.

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1));
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2) { DefaultValue = "NoName" })

13.2. ChoFallbackValue

It is the value used and set to the property when the CSV value failed to set. Fallback value only set when ErrorMode is either IgnoreAndContinue or ReportAndContinue.

Any POCO entity property can be specified with fallback value using ChoETL.ChoFallbackValueAttribute.

For dynamic object members or to override the declarative POCO object member’s fallback values, you can do through configuration as shown below.

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1));
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2) { FallbackValue = "Tom" });

13.3. FieldType

In the type less dynamic object model, the reader reads individual field value and populate them to dynamic object members in ‘string’ value. If you want to enforce the type and do extra type checking during load, you can do so by declaring the field type at the field configuration.

Listing 8.5.1 Defining FieldType

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1) { FieldType = typeof(int) });
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));

In above sample shows to define field type as ‘int’ to ‘Id’ field. This instruct the CSVReader to parse and convert the value to integer before assigning to it. This extra type safety alleviate the incorrect values being loaded to object while parsing.

13.4. Type Converters

Most of the primitive types are automatically converted and set them to the properties by CSVReader. If the value of the CSV field can’t automatically be converted into the type of the property, you can specify a custom / built-in .NET converters to convert the value. These can be either IValueConverter or TypeConverter converters.

In the dynamic object model, you can specify these converters via configuration. See below example on the approach taken to specify type converters for CSV columns

Listing 13.4.1 Specifying TypeConverters

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.FileHeaderConfiguration.HasHeaderRecord = true;
config.ThrowAndStopOnMissingField = false;

ChoCSVRecordFieldConfiguration idConfig = new ChoCSVRecordFieldConfiguration("Id", 1);
idConfig.AddConverter(new IntConverter());
config.RecordFieldConfigurations.Add(idConfig);

config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name1", 2));

In above, we construct and attach the IntConverter to ‘Id’ field using AddConverter helper method in ChoCSVRecordFieldConfiguration object.

Likewise, if you want to remove any converter from it, you can use RemoveConverter on ChoCSVRecordFieldConfiguration object.

13.5. Validations

CSVReader leverages both System.ComponentModel.DataAnnotations and Validation Block validation attributes to specify validation rules for individual CSV fields. Refer to the MSDN site for a list of available DataAnnotations validation attributes.

Listing 13.5.1 Specifying Validations

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.FileHeaderConfiguration.HasHeaderRecord = true;
config.ThrowAndStopOnMissingField = false;

ChoCSVRecordFieldConfiguration idConfig = new ChoCSVRecordFieldConfiguration("Id", 1);
idConfig.Validators = new ValidationAttribute[] { new RangeAttribute(0, 100) };
config.RecordFieldConfigurations.Add(idConfig);

config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name1", 2));

In example above, we used Range validation attribute for Id property. CSVReader performs validation on them during load based on Configuration.ObjectValidationMode is set to ChoObjectValidationMode.MemberLevel or ChoObjectValidationMode.ObjectLevel.

PS: Self validation NOT supported in Dynamic object model

14. Working with sealed POCO object

If you already have existing sealed POCO object or the object is in 3rd party library, we can use them with CSVReader.  All you need is the CSV file with header in it.

Listing 14.1 Exisiting sealed POCO Object

public sealed class ThirdPartyRec
{
    public int Id
    {
        get;
        set;
    }
    public string Name
    {
        get;
        set;
    }
}

Listing 14.2 Consuming CSV file

class Program
{
    static void Main(string[] args)
    {
        using (var stream = new MemoryStream())
        using (var reader = new StreamReader(stream))
        using (var writer = new StreamWriter(stream))
        using (var parser = new ChoCSVReader<ThirdPartyRec>(reader))
        {
            writer.WriteLine("Id,Name"); 
            writer.WriteLine("1,Carl");
            writer.WriteLine("2,Mark");
            writer.WriteLine("3,Tom");

            writer.Flush();
            stream.Position = 0;
 
            object row = null;
 
            while ((row = parser.Read()) != null)
                Console.WriteLine(row.ToString());
        }
    }
}

In this case, CSVReader reverse discover the CSV columns from the CSV file and load the data into POCO object. If the CSV file structure and POCO object matches, the load will success with populating all corresponding data to its properties. In case the property is missing for any CSV column, CSVReader silently ignores them and continue on with rest.

You can override this behavior by setting ChoCSVRecordConfiguration.ThrowAndStopOnMissingField property to false. In this case, the CSVReader will throw ChoMissingRecordFieldException exception if a property is missing for a CSV column.

15. Exceptions

CSVReader throws different types of exceptions in different situations.

  • ChoParserException – CSV file is bad and parser not able to recover.
  • ChoRecordConfigurationException – Any invalid configuration settings are specified, this exception will be raised.
  • ChoMissingRecordFieldException – A property is missing for a CSV column, this exception will be raised.

16. Tips

16.1 Multiline CSV column value

If CSV file contains column values with newline characters in it, ChoCSVReader can handle it with surrounded quotes.

Listing 16.1.1 Multiline column values in CSV file

Id,Name
1,"Tom
Cassawaw"
2,"Carl"
3,"Mark"

In above, the Id (1) has name in multiline with surrounded quotes. ChoCSVReader recognize this situation, load them propertly.

16.2 CSV column value with field separator

If CSV file contains column values with field separator (,) in it, ChoCSVReader can handle it with surrounded quotes.

Listing 16.2.1 CSV column value with separators

Id,Name
1,"Tom Cassawaw"
2,"Carl, Malcolm"
3,"Mark"

In above, the Id (2) has name with separator (,) in it. In order for ChoCSVReader recognize this situation, it must be surrounded by quotes.

16.3 CSV column value with single quotes

ChoCSVReader can read CSV column values with single quotes in it seamlessly. No surrounded quotes required.

Listing 16.3.1 CSV column value with single quotes

Id,Name
1,Tom Cassawaw
2,Carl'Malcolm
3,Mark

In above, the Id (2) has name with single quotes (‘) in it. ChoCSVReader recognize this situation, load these values successfully.

17. Using MetadataType Annotation

Cinchoo ETL works better with data annotation’s MetadataType model. It is way to attach MetaData class to data model class. In this associated class, you provide additional metadata information that is not in the data model. It roles is to add attribute to a class without having to modify this one. You can add this attribute that takes a single parameter to a class that will have all the attributes. This is useful when the POCO classes are auto generated (by Entity Framework, MVC etc) by an automatic tools. This is why second class come into play. You can add new stuffs without touching the generated file. Also this promotes modularization by separating the concerns into multiple classes.

For more information about it, please search in MSDN.

Listing 17.1 MetadataType annotation usage sample

[MetadataType(typeof(EmployeeRecMeta))]
public class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; }
}

[ChoCSVFileHeader]
[ChoCSVRecordObject(Encoding = "Encoding.UTF32", ErrorMode = ChoErrorMode.ThrowAndStop,
IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, ThrowAndStopOnMissingField = false, 
    ObjectValidationMode = ChoObjectValidationMode.MemberLevel)]
public class EmployeeRecMeta : IChoNotifyRecordRead, IChoValidatable
{
    [ChoCSVRecordField(1, FieldName = "id", ErrorMode = ChoErrorMode.ReportAndContinue )]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, 1, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }

    [ChoCSVRecordField(2, FieldName = "Name", QuoteField = true)]
    [StringLength(1)]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordFieldLoad(object target, int index, string propName, object value)
    {
        throw new NotImplementedException();
    }
 
    public bool AfterRecordLoad(object target, int index, object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordFieldLoad(object target, int index, string propName, ref object value)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordLoad(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeginLoad(object source)
    {
        throw new NotImplementedException();
    }
 
    public void EndLoad(object source)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordFieldLoadError(object target, int index, string propName, object value, Exception ex)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordLoadError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
 
    public bool TryValidate(object target, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public bool TryValidateFor(object target, string memberName, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public void Validate(object target)
    {
    }
 
    public void ValidateFor(object target, string memberName)
    {
    }
}

In above EmployeeRec is the data class. Contains only domain specific properties and operations. Mark it very simple class to look at it.

We separate the validation, callback mechanism, configuration etc into metadata type class, EmployeeRecMeta.

18. Configuration Choices

If the POCO entity class is an auto-generated class or exposed via library or it is a sealed class, it limits you to attach CSV schema definition to it declaratively. In such case, you can choose one of the options below to specify CSV layout configuration

  • Manual Configuration
  • Auto Map Configuration
  • Attaching MetadataType class

I’m going to show you how to configure the below POCO entity class on each approach

Listing 18.1 Sealed POCO entity class

public sealed class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; }
}

18.1 Manual Configuration

Define a brand new configuration object from scratch and add all the necessary CSV fields to the ChoCSVConfiguration.RecordFieldConfigurations collection property. This option gives you greater flexibility to control the configuration of CSV parsing. But the downside is that possibility of making mistakes and hard to manage them if the CSV file layout is large,

Listing 18.1.1 Manual Configuration

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.CSVFileHeaderConfiguration.HasHeaderRecord = true;
config.ThrowAndStopOnMissingField = true;
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1));
config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));

18.2 Auto Map Configuration

This is an alternative approach and very less error-prone method to auto map the CSV columns for the POCO entity class.

First define a schema class for EmployeeRec POCO entity class as below

Listing 18.2.1 Auto Map class

public class EmployeeRecMap
{
    [ChoCSVRecordField(1, FieldName = "id")]
    public int Id { get; set; }
 
    [ChoCSVRecordField(2, FieldName = "Name")]
    public string Name { get; set; } 
}

Then you can use it to auto map CSV columns by using ChoCSVRecordConfiguration.MapRecordFields method

Listing 18.2.2 Using Auto Map configuration

ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
config.MapRecordFields<EmployeeRecMap>();

foreach (var e in new ChoCSVReader<EmployeeRec>("Emp.csv", config)) 
    Console.WriteLine(e.ToString());

18.3 Attaching MetadataType class

This is one another approach to attach MetadataType class for POCO entity object. Previous approach simple care for auto mapping of CSV columns only. Other configuration properties like property converters, parser parameters, default/fallback values etc. are not considered.

This model, accounts for everything by defining MetadataType class and specifying the CSV configuration parameters declaratively. This is useful when your POCO entity is sealed and not partial class. Also it is one of favorable and less error-prone approach to configure CSV parsing of POCO entity.

Listing 18.3.1 Define MetadataType class

[ChoCSVFileHeader()]
[ChoCSVRecordObject(Encoding = "Encoding.UTF32", ErrorMode = ChoErrorMode.ReportAndContinue,
IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, ThrowAndStopOnMissingField = false, 
    ObjectValidationMode = ChoObjectValidationMode.MemberLevel)]
public class EmployeeRecMeta : IChoNotifyRecordRead, IChoValidatable
{
    [ChoCSVRecordField(1, FieldName = "id", ErrorMode = ChoErrorMode.ReportAndContinue )]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, 1, ErrorMessage = "Id must be > 0.")]
    public int Id { get; set; }

    [ChoCSVRecordField(2, FieldName = "Name", QuoteField = true)]
    [StringLength(1)]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordFieldLoad(object target, int index, string propName, object value)
    {
        throw new NotImplementedException();
    }
 
    public bool AfterRecordLoad(object target, int index, object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordFieldLoad(object target, int index, string propName, ref object value)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordLoad(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeginLoad(object source)
    {
        throw new NotImplementedException();
    }
 
    public void EndLoad(object source)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordFieldLoadError(object target, int index, string propName, object value, Exception ex)
    {
        return true;
    }
 
    public bool RecordLoadError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
 
    public bool TryValidate(object target, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public bool TryValidateFor(object target, string memberName, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public void Validate(object target)
    {
    }
 
    public void ValidateFor(object target, string memberName)
    {
    }
}

Listing 18.3.2 Attaching MetadataType class

//Attach metadata 
ChoMetadataObjectCache.Default.Attach<EmployeeRec>(new EmployeeRecMeta());

foreach (var e in new ChoCSVReader<EmployeeRec>("Emp.csv")) 
    Console.WriteLine(e.ToString()

19. LoadText Helper Method

This is little nifty helper method to parse and load CSV text string into objects.

Listing 19.1 Using LoadText method

string txt = "Id, Name\r\n1, Mark";
foreach (var e in ChoCSVReader.LoadText(txt))
   Console.WriteLine(e.ToStringEx());

20. Advanced Topics

20.1 Override Converters Format Specs

Cinchoo ETL automatically parses and converts each CSV column values to the corresponding CSV column’s underlying data type seamlessly. Most of the basic .NET types are handled automatically without any setup needed.

This is achieved through two key settings in the ETL system

  1. ChoCSVRecordConfiguration.CultureInfo – Represents information about a specific culture including the names of the culture, the writing system, and the calendar used, as well as access to culture-specific objects that provide information for common operations, such as formatting dates and sorting strings. Default is ‘en-US’.
  2. ChoTypeConverterFormatSpec – It is global format specifier class holds all the intrinsic .NET types formatting specs.

In this section, I’m going to talk about changing the default format specs for each .NET intrinsic data types according to parsing needs.

ChoTypeConverterFormatSpec is singleton class, the instance is exposed via ‘Instance’ static member. It is thread local, means that there will be separate instance copy kept on each thread.

There are 2 sets of format specs members given to each intrinsic type, one for loading and another one for writing the value, except for Boolean, Enum, DataTime types. These types have only one member for both loading and writing operations.

Specifying each intrinsic data type format specs through ChoTypeConverterFormatSpec will impact system wide. ie. By setting ChoTypeConverterFormatSpec.IntNumberStyle = NumberStyles.AllowParentheses, will impact all integer members of CSV objects to allow parentheses. If you want to override this behavior and take control of specific CSV data member to handle its own unique parsing of CSV value from global system wide setting, it can be done by specifying TypeConverter at the CSV field member level. Refer section 13.4 for more information.

NumberStyles (optional) used for loading values from CSV stream and Format string are used for writing values to CSV stream.

In this article I’ll brief about using NumberStyles for loading CSV data from stream. These values are optional. It determines the styles permitted for each type during parsing of CSV file. System automatically figures out the way to parse and load the values from underlying Culture. In odd situation, you may want to override and set the styles the way you want in order to successfully load the file. Refer the MSDN for more about NumberStyles and its values.

Listing 20.1.1 ChoTypeConverterFormatSpec Members

public class ChoTypeConverterFormatSpec
{
    public static readonly ThreadLocal<ChoTypeConverterFormatSpec> Instance = new ThreadLocal<ChoTypeConverterFormatSpec>(() => new ChoTypeConverterFormatSpec());
 
    public string DateTimeFormat { get; set; }
    public ChoBooleanFormatSpec BooleanFormat { get; set; }
    public ChoEnumFormatSpec EnumFormat { get; set; }
 
    public NumberStyles? CurrencyNumberStyle { get; set; }
    public string CurrencyFormat { get; set; }
 
    public NumberStyles? BigIntegerNumberStyle { get; set; }
    public string BigIntegerFormat { get; set; }
 
    public NumberStyles? ByteNumberStyle { get; set; }
    public string ByteFormat { get; set; }
 
    public NumberStyles? SByteNumberStyle { get; set; }
    public string SByteFormat { get; set; }
 
    public NumberStyles? DecimalNumberStyle { get; set; }
    public string DecimalFormat { get; set; }
 
    public NumberStyles? DoubleNumberStyle { get; set; }
    public string DoubleFormat { get; set; }
 
    public NumberStyles? FloatNumberStyle { get; set; }
    public string FloatFormat { get; set; }
 
    public string IntFormat { get; set; }
    public NumberStyles? IntNumberStyle { get; set; }
 
    public string UIntFormat { get; set; }
    public NumberStyles? UIntNumberStyle { get; set; }
 
    public NumberStyles? LongNumberStyle { get; set; }
    public string LongFormat { get; set; }
 
    public NumberStyles? ULongNumberStyle { get; set; }
    public string ULongFormat { get; set; }
 
    public NumberStyles? ShortNumberStyle { get; set; }
    public string ShortFormat { get; set; }
 
    public NumberStyles? UShortNumberStyle { get; set; }
    public string UShortFormat { get; set; }
}

Sample below shows how to load CSV data stream having ‘se-SE’ (Swedish) culture specific data using CSVReader. Also the input feed comes with ‘EmployeeNo’ values containing parentheses. In order to make the load successful, we have to set the ChoTypeConverterFormatSpec.IntNumberStyle to NumberStyles.AllowParenthesis.

Listing 20.1.2 Using ChoTypeConverterFormatSpec in code

static void UsingFormatSpecs()
{
    ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
    config.Culture = new System.Globalization.CultureInfo("se-SE");
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1) { FieldType = typeof(int) });
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Salary", 3) { FieldType = typeof(ChoCurrency) });
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("JoinedDate", 4) { FieldType = typeof(DateTime) });
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("EmployeeNo", 5) { FieldType = typeof(int) });
 
    ChoTypeConverterFormatSpec.Instance.IntNumberStyle = NumberStyles.AllowParentheses;
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader(reader, config))
    {
        writer.WriteLine(@"1,Carl,12.345679 kr,2017-10-10,  (5)    ");
        writer.WriteLine("2,Markl,50000 kr,2001-10-01,  6    ");
        writer.WriteLine("3,Toml,150000 kr,1996-01-25,  9    ");
 
        writer.Flush();
        stream.Position = 0;
 
        object row = null;
 
        while ((row = parser.Read()) != null)
            Console.WriteLine(row.ToStringEx());
    }
}

20.2 Currency Support

Cinchoo ETL provides ChoCurrency object to read and write currency values in CSV files. ChoCurrency is a wrapper class to hold the currency value in decimal type along with support of serializing them in text format during CSV load.

Listing 20.2.1 Using Currency members in dynamic model

static void CurrencyDynamicTest()
{
    ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1));
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Salary", 3) { FieldType = typeof(ChoCurrency) });
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader(reader, config))
    {
        writer.WriteLine("1,Carl,$100000");
        writer.WriteLine("2,Mark,$50000");
        writer.WriteLine("3,Tom,$1000");
 
        writer.Flush();
        stream.Position = 0;
 
        object rec;
        while ((rec = parser.Read()) != null)
        {
            Console.WriteLine(rec.ToStringEx());
        }
    }
}

Sample above shows how to load currency values using dynamic object model. By default, all the members of dynamic object are treated as string type, unless specified explicitly via ChoCSVFieldConfiguration.FieldType. By specifying the field type as ChoCurrency to the ‘Sa;lary’ CSV field, CSVReader loads them as currency object.

PS: The format of the currency value is figured by CSVReader through ChoRecordConfiguration.Culture and ChoTypeConverterFormatSpec.CurrencyNumberStyle.

Sample below shows how to use ChoCurrency CSV field in POCO entity class.

Listing 20.2.2 Using Currency members in POCO model

public class EmployeeRecWithCurrency
{
    public int Id { get; set; }
    public string Name { get; set; }
    public ChoCurrency Salary { get; set; }
}
 
static void CurrencyTest()
{
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader<EmployeeRecWithCurrency>(reader))
    {
        writer.WriteLine("1,Carl,$100000");
        writer.WriteLine("2,Mark,$50000");
        writer.WriteLine("3,Tom,$1000");
 
        writer.Flush();
        stream.Position = 0;
 
        object rec;
        while ((rec = parser.Read()) != null)
        {
            Console.WriteLine(rec.ToStringEx());
        }
    }
}

20.3 Enum Support

Cinchoo ETL implicitly handles parsing of enum column values from CSV files. If you want to fine control the parsing of these values, you can specify them globally via ChoTypeConverterFormatSpec.EnumFormat. Default is ChoEnumFormatSpec.Value

FYI, changing this value will impact system wide.

There are 3 possible values can be used

  1. ChoEnumFormatSpec.Value – Enum value is used for parsing.
  2. ChoEnumFormatSpec.Name – Enum key name is used for parsing.
  3. ChoEnumFormatSpec.Description – If each enum key is decorated with DescriptionAttribute, its value will be use for parsing.

Listing 20.3.1 Specifying Enum format specs during parsing

public enum EmployeeType
{
    [Description("Full Time Employee")]
    Permanent = 0,
    [Description("Temporary Employee")]
    Temporary = 1,
    [Description("Contract Employee")]
    Contract = 2
}

static void EnumTest()
{
    ChoTypeConverterFormatSpec.Instance.EnumFormat = ChoEnumFormatSpec.Description;
 
    ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1) { FieldType = typeof(int) });
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Salary", 3) { FieldType = typeof(ChoCurrency) });
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("JoinedDate", 4) { FieldType = typeof(DateTime) });
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("EmployeeType", 5) { FieldType = typeof(EmployeeType) });
 
    ChoTypeConverterFormatSpec.Instance.IntNumberStyle = NumberStyles.AllowParentheses;
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader(reader, config))
    {
        writer.WriteLine("1,Carl,12345679,01/10/2016,Full Time Employee");
        writer.WriteLine("2,Mark,50000,10/01/1995,Temporary Employee");
        writer.WriteLine("3,Tom,150000,01/01/1940,Contract Employee");
 
        writer.Flush();
        stream.Position = 0;
 
        object row = null;
 
        while ((row = parser.Read()) != null)
            Console.WriteLine(row.ToStringEx());
    }
}

20.4 Boolean Support

Cinchoo ETL implicitly handles parsing of boolean CSV column values from CSV files. If you want to fine control the parsing of these values, you can specify them globally via ChoTypeConverterFormatSpec.BooleanFormat. Default value is ChoBooleanFormatSpec.ZeroOrOne

FYI, changing this value will impact system wide.

There are 4 possible values can be used

  1. ChoBooleanFormatSpec.ZeroOrOne – ‘0’ for false. ‘1’ for true.
  2. ChoBooleanFormatSpec.YOrN – ‘Y’ for true, ‘N’ for false.
  3. ChoBooleanFormatSpec.TrueOrFalse – ‘True’ for true, ‘False’ for false.
  4. ChoBooleanFormatSpec.YesOrNo – ‘Yes’ for true, ‘No’ for false.

Listing 20.4.1 Specifying boolean format specs during parsing

static void BoolTest()
{
    ChoTypeConverterFormatSpec.Instance.BooleanFormat = ChoBooleanFormatSpec.ZeroOrOne;
 
    ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1) { FieldType = typeof(int) });
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Salary", 3) { FieldType = typeof(ChoCurrency) });
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("JoinedDate", 4) { FieldType = typeof(DateTime) });
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Active", 5) { FieldType = typeof(bool) });
 
    ChoTypeConverterFormatSpec.Instance.IntNumberStyle = NumberStyles.AllowParentheses;
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader(reader, config))
    {
        writer.WriteLine("1,Carl,12345679,01/10/2016,0");
        writer.WriteLine("2,Mark,50000,10/01/1995,1");
        writer.WriteLine("3,Tom,150000,01/01/1940,1");
 
        writer.Flush();
        stream.Position = 0;
 
        object row = null;
 
        while ((row = parser.Read()) != null)
            Console.WriteLine(row.ToStringEx());
    }
}

20.5 DateTime Support

Cinchoo ETL implicitly handles parsing of datetime CSV column values from CSV files using system Culture or custom set culture. If you want to fine control the parsing of these values, you can specify them globally via ChoTypeConverterFormatSpec.DateTimeFormat. Default value is ‘d’.

FYI, changing this value will impact system wide.

You can use any valid standard or custom datetime .NET format specification to parse the datetime CSV values from the file.

Listing 20.5.1 Specifying datetime format specs during parsing

static void DateTimeTest()
{
    ChoTypeConverterFormatSpec.Instance.DateTimeFormat = "MMM dd, yyyy";
 
    ChoCSVRecordConfiguration config = new ChoCSVRecordConfiguration();
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Id", 1) { FieldType = typeof(int) });
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Name", 2));
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Salary", 3) { FieldType = typeof(ChoCurrency) });
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("JoinedDate", 4) { FieldType = typeof(DateTime) });
    config.RecordFieldConfigurations.Add(new ChoCSVRecordFieldConfiguration("Active", 5) { FieldType = typeof(bool) });
 
    ChoTypeConverterFormatSpec.Instance.IntNumberStyle = NumberStyles.AllowParentheses;
 
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader(reader, config))
    {
        writer.WriteLine(@"1,Carl,12345679,""Jan 01, 2011"",0");
        writer.WriteLine(@"2,Mark,50000,""Sep 23, 1995"",1");
        writer.WriteLine(@"3,Tom,150000,""Apr 10, 1999"",1");
 
        writer.Flush();
        stream.Position = 0;
 
        object row = null;
 
        while ((row = parser.Read()) != null)
            Console.WriteLine(row.ToStringEx());
    }
}

Sample above shows how to parse custom datetime CSV values from CSV file.

Note: As the datetime values contains CSV seperator, it is given with double quotes to pass the parsing.

21. Fluent API

CSVReader exposes few frequent to use configuration parameters via fluent API methods. This will make the programming of parsing of CSV files quicker.

21.1 WithDelimiter

This API method sets the CSV field seperator on CSVReader.

static void QuickDynamicTest()
{
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader(reader).WithDelimiter(","))
    {
        writer.WriteLine("1,Carl,1000");
        writer.WriteLine("2,Mark,2000");
        writer.WriteLine("3,Tom,3000");
 
        writer.Flush();
        stream.Position = 0;
 
        object rec;
        while ((rec = parser.Read()) != null)
        {
            Console.WriteLine(rec.ToStringEx());
        }
    }
}

21.2 WithFirstLineHeader

This API method flags the CSV file contains first row as header or not. Optional bool parameter specifies the first row header or not. Default is true.

static void QuickDynamicTest()
{
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader(reader).WithDelimiter(",").WithFirstLineHeader())
    {
        writer.WriteLine("Id,Name,Salary");
        writer.WriteLine("1,Carl,1000");
        writer.WriteLine("2,Mark,2000");
        writer.WriteLine("3,Tom,3000");
 
        writer.Flush();
        stream.Position = 0;
 
        object rec;
        while ((rec = parser.Read()) != null)
        {
            Console.WriteLine(rec.ToStringEx());
        }
    }
}

21.3 WithFields

This API method specifies the list of CSV fields to be considered for parsing and loading. Other fields in the CSV file will be discarded. This call re-initializes with the specified columns.

In dynamic object model, all the CSV columns will be created and parsed as string type.

static void QuickDynamicTest()
{
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader(reader).WithFirstLineHeader().WithFields("Id", "Name"))
    {
        writer.WriteLine("Id,Name,Salary");
        writer.WriteLine("1,Carl,1000");
        writer.WriteLine("2,Mark,2000");
        writer.WriteLine("3,Tom,3000");
 
        writer.Flush();
        stream.Position = 0;
 
        object rec;
        while ((rec = parser.Read()) != null)
        {
            Console.WriteLine(rec.ToStringEx());
        }
    }
}

21.4 WithField

This API method used to add CSV column with specific date type. This method helpful in dynamic object model, by specifying each and individual CSV column with appropriate datatype.

static void QuickDynamicTest()
{
    using (var stream = new MemoryStream())
    using (var reader = new StreamReader(stream))
    using (var writer = new StreamWriter(stream))
    using (var parser = new ChoCSVReader(reader).WithFirstLineHeader().WithField("Id", typeof(ini).WithField("Name", typeof(string))
    {
        writer.WriteLine("Id,Name,Salary");
        writer.WriteLine("1,Carl,1000");
        writer.WriteLine("2,Mark,2000");
        writer.WriteLine("3,Tom,3000");
 
        writer.Flush();
        stream.Position = 0;
 
        object rec;
        while ((rec = parser.Read()) != null)
        {
            Console.WriteLine(rec.ToStringEx());
        }
    }
}

iSkeddy – FTP_PUT_FILE task

This task used to download a file from remote FTP site.

Image 1 FTP_PUT_FILE parameters

FTP_PUT_FILE

Here are the list of parameters supported by this task

  • Request URI string – The URI that identifies the Internet resource / file.
  • Source File Path – Absolute source file path.
  • EnableSsl – Gets or sets a Boolean that specifies that an SSL connection should be used.
  • KeepAlive – Gets or sets a Boolean value that specifies whether the control connection to the FTP server is closed after the request completes.
  • Password – The password for the user name associated with the credentials.
  • Proxy – Gets or sets the proxy used to communicate with the FTP server. Specify ‘IE’ if you have IE settings for proxy to use.
  • Use Binary – Gets or sets a System.Boolean value that specifies the data type for file transfers.
  • Use Passive – Gets or sets a Boolean value that specifies whether the control connection to the FTP server is closed after the request completes.
  • User Name – The user name associated with the credentials.

iSkeddy – FTP_GET_FILE task

This task used to download a file from remote FTP site.

Image 1 FTP_GET_FILE parameters

FTP_GET_FILE

Here are the list of parameters supported by this task

  • Request URI string – The URI that identifies the Internet resource / file.
  • Destination File Path – Absolute destination file path.
  • EnableSsl – Gets or sets a Boolean that specifies that an SSL connection should be used.
  • KeepAlive – Gets or sets a Boolean value that specifies whether the control connection to the FTP server is closed after the request completes.
  • Password – The password for the user name associated with the credentials.
  • Proxy – Gets or sets the proxy used to communicate with the FTP server. Specify ‘IE’ if you have IE settings for proxy to use.
  • Use Binary – Gets or sets a System.Boolean value that specifies the data type for file transfers.
  • Use Passive – Gets or sets a Boolean value that specifies whether the control connection to the FTP server is closed after the request completes.
  • User Name – The user name associated with the credentials.

iSkeddy – FTP_FILE_WATCH task

This task monitors existence of file at the remote FTP site.

Image 1 FTP_FILE_WATCH parameters

FTP_FILE_WATCH

Here are the list of parameters supported by this task

  • Request URI string – The URI that identifies the Internet resource / file.
  • EnableSsl – Gets or sets a Boolean that specifies that an SSL connection should be used.
  • KeepAlive – Gets or sets a Boolean value that specifies whether the control connection to the FTP server is closed after the request completes.
  • Password – The password for the user name associated with the credentials.
  • Proxy – Gets or sets the proxy used to communicate with the FTP server. Specify ‘IE’ if you have IE settings for proxy to use.
  • User Name – The user name associated with the credentials.

Cinchoo – Abortable long running aync task

Introduction

In .NET, there are number of ways you can start asynchronous long running tasks by using Thread / Task / ThreadPool. Every approach has pros and cons to it. Please refer MSDN for more information about them. In this article I’m going to elobarate about method exists in Cinchoo framework where you can start abortable long running task asnychrously.

First, each windows application has a single main thread it uses to render the window / execute main task, process events and execute custom code. In a single threaded environment, any long running tasks will block the main thread. In particlular it will block the main UI thread and the application becomes unresponsive in Windows application.

Cinchoo simplifies the model to execute a long running tasks asynchronously and gives control to the caller to abort it anytime. Besides the standard behaviour, it offers few useful features like number of retries in case of failure, timeout period via the API. Alright lets cut to the chase on to how to use them using the code.

Download the Latest Cinchoo Binary Here. (Nuget Command: Install-Package Cinchoo)

Using the Code

The API is exposed via ChoActionEx class. The signature of the API looks below

public static class ChoActionEx
{
    public static ChoAbortableAsyncResult RunAsync(this Action action, ChoAbortableAsyncCallback callback = null, object state = null, int timeout = -1, int maxNoOfRetry = 0, int sleepBetweenRetry = 5000);
}

Where

  • action – A long running task method.
  • callback – References a method to be called when a corresponding asynchronous operation completes.
  • state – A user-defined object that qualifies or contains information about an asynchronous operation.
  • timeout – A time-out value in milliseconds. -1, infinite.
  • maxNoOfRetry – Maximum number of retry attempt made in case of failure.
  • sleepBetweenRetry – Number of milliseconds to sleep between retry. Default is 5000 ms.

This API method returns ChoAbortableAsyncResult object. This encapsulates the results of an asynchronous operation on a delegate. It exposes the below members

  • AsyncState – Gets the object provided in the ‘state’ parameter of a RunAsync method call.
  • AsyncWaitHandle – Gets a WaitHandle that encapsulates Win32 synchronization handles, and allows the implementation of various synchronization schemes.
  • CompletedSynchronously – Gets a value indicating whether the RunAsync call completed synchronously.
  • IsAborted – Gets a value indicating whether the server has aborted the call.
  • IsCompleted – Gets a value indicating whether the server has completed the call.
  • IsRetryAttempt – Gets a value indicating whether the server has retried the call.
  • IsTimeout – Gets a value indicating whether the method has completed with the timeout.
  • Result – Gets the result value from the async call if any.
  • RetryCount – Gets a value of the number of retry attempt made by the server.
  • Exception – Gets any exception captured during task execution.
  • CanContinue – Gets or Sets where to continue the retry.
  • EndInvoke() – Retrieves the return value of the asynchronous operation. If the asynchronous operation has not been completed, this function will block until the result is available.
  • Abort() – Abort the currently executing asynchrounous call.

Sample #1

Below sample shows how to execute a method asynchrounously, wait for it to complete.

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Main thread: {0}", Thread.CurrentThread.ManagedThreadId);
        Console.WriteLine();
 
        //Run method async, wait for it to complete
        Console.WriteLine("TEST #1: Run method, wait for it to complete...");
        ChoAbortableAsyncResult r = ChoActionEx.RunAsync(LongRunningTask);
        Console.WriteLine("Waiting for worker thread to complete.");
        r.EndInvoke();
        Console.WriteLine();
    }
 
    private static void LongRunningTask()
    {
        Console.WriteLine("Starting task... (Sleeping for 10 secs)");
        Console.WriteLine("Worker thread: {0}", Thread.CurrentThread.ManagedThreadId);
        Thread.Sleep(10 * 1000);
        Console.WriteLine("Task completed.");
    }
}

Sample #2

Below sample shows how to execute a method asynchrounously, abort it after 5 secs.

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Main thread: {0}", Thread.CurrentThread.ManagedThreadId);
        Console.WriteLine();

        //Run method async, abort it after 5 secs
        Console.WriteLine("TEST #2: Run method, abort after 5 secs...");
        ChoAbortableAsyncResult r1 = ChoActionEx.RunAsync(LongRunningTask);
        Console.WriteLine("Waiting for 5 secs...");
        Thread.Sleep(5000);
        Console.WriteLine("Aborting working thread.");
        r1.Abort();
        Console.WriteLine();
    }
 
    private static void LongRunningTask()
    {
        Console.WriteLine("Starting task... (Sleeping for 10 secs)");
        Console.WriteLine("Worker thread: {0}", Thread.CurrentThread.ManagedThreadId);
        Thread.Sleep(10 * 1000);
        Console.WriteLine("Task completed.");
    }
}

Sample #3

Below sample shows how to execute a method asynchrounously, timeout after 5 secs.

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Main thread: {0}", Thread.CurrentThread.ManagedThreadId);
        Console.WriteLine();

        //Run method async, with timeout 5 secs
        Console.WriteLine("TEST #3: Run method with 5 secs timeout...");
        ChoAbortableAsyncResult r2 = ChoActionEx.RunAsync(LongRunningTask, null, null, 5000);
        Console.WriteLine("Waiting for worker thread to complete or timeout.");
        try
        {
            r2.EndInvoke();
        }
        catch (Exception e)
        {
            Console.WriteLine(e.Message);
        }
        Console.WriteLine();
    }
 
    private static void LongRunningTask()
    {
        Console.WriteLine("Starting task... (Sleeping for 10 secs)");
        Console.WriteLine("Worker thread: {0}", Thread.CurrentThread.ManagedThreadId);
        Thread.Sleep(10 * 1000);
        Console.WriteLine("Task completed.");
    }
}

Sample #4

Below sample shows how to execute a method asynchrounously, retry couple of attempts.

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Main thread: {0}", Thread.CurrentThread.ManagedThreadId);
        Console.WriteLine();

        //Run a exception thrown method async 
        Console.WriteLine("TEST #4: Run method with 2 retries...");
        ChoAbortableAsyncResult r3 = ChoActionEx.RunAsync(LongRunningTaskWithException, null, null, -1, 2, 5000);
        Console.WriteLine("Waiting for worker thread to complete.");
        try
        {
            r3.EndInvoke();
        }
        catch (Exception e)
        {
            Console.WriteLine(e.Message);
        }
        Console.WriteLine();
    }
    
    private static void LongRunningTaskWithException()
    {
        Console.WriteLine("Starting task... (Sleeping for 10 secs)");
        Console.WriteLine("Worker thread: {0}", Thread.CurrentThread.ManagedThreadId);
        Thread.Sleep(10 * 1000);
        throw new ApplicationException("Test task exception.");
    }
}

Sample #5

Below sample shows how to execute a method asynchrounously, cancel it in the callback.

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Main thread: {0}", Thread.CurrentThread.ManagedThreadId);
        Console.WriteLine();

        Console.WriteLine("TEST #5: Run method with 2 retries, but cancel the retry on callback...");
        ChoAbortableAsyncResult r5 = ChoActionEx.RunAsync(LongRunningTaskWithException, (t) =>
        {
            ChoAbortableAsyncResult t1 = t as ChoAbortableAsyncResult;
            Console.WriteLine("Canceling the task...");
            if (t1.Exception != null)
                t1.CanContinue = false;
        }, null, -1, 2, 5000);
        try
        {
            r5.EndInvoke();
        }
        catch (Exception e)
        {
            Console.WriteLine(e.Message);
        }
        Console.WriteLine();
    }
    
    private static void LongRunningTaskWithException()
    {
        Console.WriteLine("Starting task... (Sleeping for 10 secs)");
        Console.WriteLine("Worker thread: {0}", Thread.CurrentThread.ManagedThreadId);
        Thread.Sleep(10 * 1000);
        throw new ApplicationException("Test task exception.");
    }
}

iSkeddy – Enterprise workload automation and scheduling system

Download binary

1. Overview

iSkeddy is an enterprise workload automation and scheduling software. It combines many features. With simple to use GUI client, through which scheduling can be set to your jobs run on any schedule, recurring or event. Work on reliably, even during unexpected failure. With the expanding IT organizations, managing number of applications, databases and technologies that are critical to the successful operations of the organization. With iSkeddy, you can develop and rollout end-to-end workflows faster and more reliably than using custom scripts or “closed” scheduling systems.

PS: iSkeddy developed with Cinchoo, an application framework for .NET

2. Features

iSkeddy lets you run your jobs on simple or complex recurring schedules. Below are some of the key features offered by iSkeddy

  • Scheduler – iSkeddy scheduler lets you to create tasks that could reliably run them on recurring schedule or at some point in future.
  • Automation – It provides a graphical user interface and a single point of control for definition and monitoring of background tasks executions in a distributed network of computers.
  • Easy User Interface – iSkeddy dashboard gives nice user interface to manage and monitor tasks right from your desk.
  • Extended Logging – Collect verbose logs from the task run and view them from dashboard.
  • Auditing for compliance – iSkeddy offers full support for audit and policies. It records and stores the each action initiated by user.
  • Simple Workflow/Job setup – Group multiple tasks into logical sequences, link them your way.
  • Load Balancing – Tasks workloads are evenly distributed among clusters.
  • Resiliency – Automatically fail over to secondary node when primary goes down, eliminate the down time seconds to none.
  • Tasks Library – Number of preloaded tasks to get your work started.
  • Extensions – Easily extendable to create and use your own custom tasks for your needs.

3. Components

iSkeddy is a client-server system. Very simple to setup and run, zero configuration required. Just download the setup.exe and install it on your system. It will put two components in C:\Program Files (x86)\iSkeddy folder

  • iSkeddy client (iSkeddy.exe)
  • iSkeddy server (iSkeddyServer.exe)

Simply launch both iSkeddyServer.exe and iSkeddy.exe, you are ready to setup and schedule any type of tasks within few seconds.

3.1 iSkeddy Client

It is simple easy to use interface component, lets you setup, manage and monitor tasks/jobs. At launch, the iSkeddy client automatically try to discover and connect to the available iSkeddy server if the UDP protocol is turned on. If UDP is not turned on, you can choose to connect to iSkeddy server via TCP/IP protocol using ‘Server Settings’ window. You can get to this window by using ‘Connect’ button on the main window.

Image 3.1 iSkeddy Server Settings Window

ConnectWnd1

The above settings are

  • ControllerIPAddress – In TCP/IP mode, you will have to specify the iSkeddy controller IP address.
  • IsPeer – N/A
  • TurnOnUDP – Turn on / off UDP.
  • UDPAddress – In case of UDP turned on, specify the UDP address.
  • UDPPort – UDP port.

The main window of iSkeddy client looks as below

Image 3.2 iSkeddy Main Window

The main window consists of two panes. On the left side pane, it lists all the tasks grouped together in tree view format. On the right side pane, it shows the selected jobs running statuses in grid view format.

Task is smallest operation that serves as unit of work. In iSkeddy, you can setup a task with specific custom name. A task can set to one or more schedules, by which you can set them to run different schedule/calendar based on your needs. Each task must be tied to workflow definition. Workflow is the series of activities that are necessary to complete a task. iSkeddy comes with standard workflow definition ready to use with your tasks. You can extend iSkeddy to define your own workflow definition and use them with your tasks as well. For each task run, a unique batch id will be generated and associated with it.

Let me describe each of the column displayed in the grid…

  • TaskName – A unique name assigned to a task.
  • ScheduleName – A task schedule name. Task can have one or more schedules. Each schedule will be associated with a unique name.
  • JobName – A job name is the combined name of TaskName and ScheduleName. Ex. ‘1_F1_FILE_WATCH’ task with ‘S1’ schedule, the job name will be derived as ‘1_F1_FILE_WATCH_S1’.
  • BatchId – A unique identifier generated and associated with each task run.
  • Status – A running status of task’s run. Possible states are
    • SUCCESS – After successful run, SUCCESS will be set.
    • FAILED – After failed run, FAILED status will be set.
    • ACTIVE – A job must be active in order to run or schedule.
    • SCHEDULED – A job is put on queue, waiting to be run by scheduler.
    • INACTIVE – Inactive tasks will be excluded from scheduler.
    • RUNNING – A current running tasks will be set to RUNNING state
    • STOPPED – Graceful way to stop the task by user, will be set as STOPPED status.
    • ABORTED – Abnormal terminated tasks are set as ABORTED status.
    • TIMEDOUT – A task is not completed within the specified period of time, it will terminated by iSkeddy and set to TIMEDOUT status.
  • Parameters – Task run time parameters, in key-value pairs.
  • Message – Running status message of the task.
  • StartTime – Start time of the task run.
  • EndTime – End time of the task run.
  • Duration – Elapsed time of the task run.
  • NextRun – Next scheduled run date time of the task, If the task dependent on other tasks with conditions, it will display the condition.
  • Exit – Exit code from the task run.
  • Retry – Retry attempt count.
  • Machine – Machine name where task will run.
  • User – It is the user who requested task action. SYSTEM, if the task is managed by iSkeddy.

On top of the Main Window, you will notice few buttons

  • Refresh – It is used to reconnect iSkeddy client to server, in case of any unexpected disconnect happens.
  • View Clusters – In the cluster setup, where more that one iSkeddy server nodes are hosted, this will open a window to view all the available active nodes.

Image 3.3 iSkeddy Cluster Nodes Window

Cluster

 

  • Holiday Calendar – This will launch the holiday calendar window, where you can manage the holiday calendars. Holiday calendars define the set of statutory (civic) holidays that tasks are entitled to take off. You can associate a calendar to a task, the scheduler will remember not to run the task on those holidays. The holiday calendar window looks as below. Will discuss in detail about it later.

Image 3.4 iSkeddy Holiday Calendar Window

HolidayCalendar

 

3.1.1 Manage Tasks

Tree-ContextMenuIn this section, I’ll elaborate about creating, managing tasks/jobs using iSkeddy client program. There are two types of nodes can be created

  • Group – It is logical grouping of tasks managed under a group.
  • Task –  It is an actual task unit.

When you right-click on any node on the tree, you will have context menu shown in Image 3.1.1.1

  • Add Group – Add a new Group.
  • Add Task – Add a new Task.
  • Remove – Remove Group / Task.
  • Edit – Edit a Group / Task.
  • Copy – Copy a Task.
  • Paste – Paste a Task.
  • Move – Move a Task from one group to another.
  • Export – Export a Task/Group.
  • Import – Import a Task/Group.
3.1.1.1 Add / Edit Task Group

After you click ‘Add Group’ or ‘Edit’ from the context menu, you will be prompted with the below window to manage the groups.

Image 3.1.1.1.1 Task Group Editor

AddGroup

In the above window, you will have given Task Group Name, which is unique and mandatory value. Besides name, you can provide optional comments for the task group.

3.1.1.2 Add / Edit Task

After you choose ‘Add Task’ or ‘Edit’ from the context menu, you will be prompted with the below window to manage a task

Image 3.1.1.2.1 Task Editor

TaskEditThere are many parameters to configure a task using this window.

  • Task Info – Basic task related parameters are set here.
  • Schedules – List of schedules associated with the task.
  • Schedule Parameters – Task schedule parameters are set here
  • TimeZone – Task’s Time Zone information

Task Info

In this section, the basic task parameters are specified. Some of them are

  • Task Name – Task name, it is a unique name to identify the task.
  • Machine – Machine name or Cluster Name where the task to be run.
  • WF Name – Workflow name of the task. Workflow is task execution plan. There are number of ready to use workflows comes with iSkeddy. If you click ‘Choose’ button, it will open the workflow selection windows as below

Image 3.1.1.2.2 Workflow Selection Window

WorkflowSelector

 

  • Parameters – Run parameters of the task. List of parameters needed to run the task. It varies to each type of task. Values can be constant values or expression values.

Image 3.1.1.2.3 Task Parameters Editor

RUN_PROGRAM

 

Tip:  If a parameter value contains whitespace, it must be surrounded by single quotes (‘) to consider as part of the value.

Tip: The parameter values can be constant values or it can contain expressions (ex: TODAY, TIME etc with formats). Will discuss more about it later.

Schedules

In this section, task schedules will be managed. A task may have one or more schedules with the schedule conditions. The list box shows the list of schedules. In order to create a new schedule, type a name in the textbox and click ‘ADD NEW’. To delete a schedule, select the schedule from the listbox and click ‘DELETE’. In case you wanted to retain the schedule definition but wanted to exclude it from the scheduler, you can do so by unchecking ‘Enabled’ checkbox.

Schedule Parameters

In this section, each task schedule parameters can be specified. There are a huge list of parameters can be configured based on your needs. A task must have atleast one schedule in order to be scheduled. It can have one or more schedules with the different schedule conditions. Here are the details of the each parameter

  • StartDate – Scheduled task run start date. Must specify
  • EndDate – Scheduled task run end date. If left empty, the task will scheduled forever.
  • StartTime – Start time on a schedule day window, the task is scheduled to start.
  • EndTime – End time on a schedule day window, the task is scheduled to end.
  • ScheduleType – Schedule Type (Possible values are DAILY, WEEKLY, MONTHLY, ONCE)
  • T1 – This denotes number of days before or after the task should run when it falls on one of the holiday calendar dates. Only applicable to Weekly / Monthly schedules.
  • Daily – Specify how many days once the task should run.
  • Weekly – Specify the week days or number of weeks once the task should run.
  • Monthly – Specify the days of the month the task should run.
  • Repeat – Specify the task to run repeatedly every specified minutes / hours on a schedule day.
  • Holiday Calendar – Optional Holiday calendar attached to the task.
    • Checked – The task will NOT run on the holiday calendar days specified in the calendar.
    • Unchecked – The task will run only on those days specified in the calendar.
  • Daylight Savings Options
    • Preserve Hours of the Day across DST – This property will fire of the task to always at the same time of day regardless of Daylight Savings Time.
    • Skip Day if the hour not exists – If the hour of the day does not exists on a given day, the day will be skipped and task will be advanced to next scheduled day.
  • Other / Dependencies – Define dependency rules to fire the task. The button glows green when there is dependency rules are defined for the task. There are number of operators and functions available to use when you define the rules. JobName is in the format of TaskName/ScheduleName. FireAfter option lets to instruct the system to start the job after specified number of minutes.

Image 3.1.1.2.3 Task Dependencies Editor

Depends

 

  • Comments – Short description of the schedule for your reference.

3.1.2 Tasks Commands

In this section, I’m going elaborate about sending task commands from iSkeddy dashboard. From iSkeddy client, you can monitor, start, stop, view status, log info of any job from your desktop.

Image 3.1.2.1 iSkeddy client main window

iSkeddy-MainWindow4

 

 

Here are the available job commands

  • Start – Start a task manually
  • Resume – In case, job failed abnormally, you can resume it by using this command. This is useful when the job has resumable capability.
  • Stop – Stop the job gracefully.
  • Abort – Abort the job.
  • Activate – Activate a job. In order for a job to be scheduled or start, it must be activated first.
  • Inactivate – Inacivate a job. This will put the job on hold from scheduling as well as from starting.
  • Activate All – Activate group of related jobs. Related jobs are jobs dependent of each other.
  • Inactivate All – Inactivate group of related jobs.
  • Open Log File – Open the log file of job.

Here are the details of each column of the grid on the main window

  • Job Name – The job name (Task Name / Schedule Name).
  • Workflow Name – The workflow name.
  • Batch Id – Unique batch id for each run.
  • Status – Job status of each run.
  • Parameters – Running parameters of the job.
  • Message – Last output message of the job run.
  • Start Time – Start time of the job.
  • End Time – End time of the job.
  • Duration – Total duration of the job run.
  • Next Run – Next scheduled run date time if independent task. For dependent task, it will display the dependency expression.
  • Exit – Exit code of the job run.
  • Retry – Total number of retry happened for job.
  • Machine – Machine where the job ran.
  • User – User initiated the task action command. SYSTEM if the system schedule and run it.
  • Modified Date – Last modified date time.

3.2 iSkeddy Server

It is a server component of iSkeddy system. It built with scheduler (controller), automation functionalities. You can setup and configure iSkeddy server in multiple fashions like single instance node, multiple cluster nodes etc. In an environment, you must configure one scheduler and optional multiple automation cluster nodes.

In a simple model, just launch iSkeddy server. This will run both Scheduler (controller) and Automation functionalities in one instance.

The server can be setup to use either TCP/IP or UDP protocols. Based on your needs and protocol availabilities you can setup iSkeddy server with either TCP/IP or UDP protocal. UDP is preferred, it gives the flexibility of auto fail over to secondary node when primary goes down.

iSkeddy server uses Sqlite as internal data store. It can be replaced with using other databases (SqlServer, Oracle etc) by writing adapters.

Only one instance of iSkeddy server per machine allowed. iSkeddy server can be run as Console application or setup as Windows service.

Console Mode

Launching iSkeddyServer.exe, it will run as console mode. At launch, it pops up the below setup window.

Image 3.2.1 Server Settings Window

ConnectWnd1

In here, you can configure the protocols and endpoints of iSkeddy server

  • TurnOnUDP – If checked, UDP protocol is used. Otherwise TCP/IP is used for connection.
  • ControllerIPAddress – If it is automation node, specify the IP address of the controller (scheduler). Only applicable if TCP/IP is chosen protocol. If UDP is choosen, each automation node automatically discovers the Controller (scheduler) node.
  • IsPeer – In cluster environment, any automation nodes should be setup with this option checked. It will launch the node as automation node.
  • UDPAddress – UDP Address. Applicable only if UDP protocol is selected.
  • UDPPort – UDP port.

Here are the list of command line iSkeddy server options available to use

Listing 3.2.1 iSkeddy server command line parameters

C:iSkeddy>iSkeddyServer.exe /??
iSkeddyServer [Version 2.0.0.1]
Copyright c Cinchoo Inc 2016

Enterprise workload automation, scheduling server

ISKEDDYSERVER [/d] [/c:] [/p] [/peer] [/u] [/setup] [/s]

        /d      Debug mode
        /c      Cluster Name
        /p      Private Node
        /peer   Run as peer server
        /u      Turn on users and policies
        /setup  Setup / Refresh database.
        /s      Silent, do not prompt connection dialog-box at startup.

From above,

  • /d – Debug mode is used to instruct the server to switch to debug mode. It is useful to run and step through the task source code.
  • /c – Cluster Name
  • /p – Private Node, instruct the controller (scheduler) do not participate in workload automation, It means the node is dedicated to scheduling only, No tasks will be scheduled to run on this node.
  • /peer – Run the instance as automation node. It means the node is participated in workload automation.
  • /u – Reserved.
  • /setup – Setup mode, run once after installing iSkeddy on a machine.
  • /s – Silent, do not show the connection dialog when running in console mode.

Besides the above set of command line arguments, iSkeddy server exposes another set of command line arguments for debug a specific workflow

Listing 3.2.2 iSkeddy server run workflow command line parameters

C:\iSkeddy>iSkeddyServer.exe /?
iSkeddyServer [Version 2.0.0.1]
Copyright c Cinchoo Inc 2016

Enterprise workload automation, scheduling server

The syntax of this command is:

ISKEDDYSERVER [RUN_WF]

Where RUN_WF switch instruct the iSkeddy server to run a specific task in debug mode.

Sample command is

iSkeddyServer.exe "RUN_WF" "/t:4_FI_TEST_EXIT_CODE" "/s:S1" "/b:-922087807" "/w:C:\Personal\Cinchoo Source\2013-11-05\ChoWorkflowServer\ChoWorkflowServer\bin\Debug\WorkflowDefs\SHELL\RUN_PROGRAM.etl" "/u:RAJ" "/p:PROGRAM_NAME=C:\Users\raj\test.bat;DOMAIN=;USER_NAME=;PASSWORD=;CMD_ARGS=;SILENT=True;TIMEOUT=-1;WAIT_FOR_EXIT=True;WORKING_DIRECTORY=C:\Users\raj;USE_SHELL_EXECUTE=False" "/to:30000" "/sr:0" "/mr:0" "/d"

iSkeddy server outputs these details in the log file for you to grab them.

Windows Service Mode

You can set iSkeddy server as windows service, lets you run the server in the background, start automatically at the system startup etc by installing as Windows Service. Here is how you can install and start the iSkeddyServer as service

Listing 3.2.3 Install iSkeddy server as service

C:\iSkeddy>iSkeddyServer.exe /@I
iSkeddyServer [Version 2.0.0.1]
Copyright c Cinchoo Inc 2016


Enterprise workload automation, scheduling server

[SC] CreateService SUCCESS

[SC] ChangeServiceConfig SUCCESS

To start the service, either you can issue the below command in the command line or you can go to services console, start the service from there.

Listing 3.2.4 Start iSkeddy server service

C:\iSkeddy>iSkeddyServer.exe /@S
iSkeddyServer [Version 2.0.0.1]
Copyright c Cinchoo Inc 2016


Enterprise workload automation, scheduling server

For more service related commands, please run the below command

Listing 3.2.5 iSkeddy server service control parameters

C:iSkeddy>iSkeddyServer.exe /@?
iSkeddyServer [Version 2.0.0.1]
Copyright c Cinchoo Inc 2016


Enterprise workload automation, scheduling server

ISKEDDYSERVER [/@SN:] [/@DN:] [/@IN:] [/@SD:] [/@I] [/@U] [/@S] [/@T] [/@P] [/@C] [/@E:] [/@SP:]

        /@SN    Service Name.
        /@DN    Display Name.
        /@IN    Instance Name.
        /@SD    Service Description.
        /@I     Install Service.
        /@U     Uninstall Service.
        /@S     Start Service.
        /@T     Stop Service.
        /@P     Pause Service.
        /@C     Continue Service.
        /@E     Execute Command.
        /@SP    Command Line Parameters.

Tip:

If you want to run iSkeddyServer in different account/credentials, you can do so by opening ChoServiceInstallerSettings.xml configuration file under Config folder and specify the account details.

Image 3.2.2 iSkeddy server log folder screenshot

iSkeddyServerLog

 

iSkeddyServer logs are created under C:\Program Files (x86)\iSkeddy\iSkeddyServer\Logs. Every group of tasks run associated with new batch id, if there are dependent on each other. Or if it is independent task, a unique batch id assigned to it for each run.

For each such run, a separate log folder created in the name of batch id. All jobs log files are created under there. For example, a batch id 695455002, below image shows 2 related jobs logs are created under there.

Image 3.2.3 iSkeddy server log sub-folder screenshot

iSkeddyServerLog1

It simplifies and well organizes the log files, so that you can easily track them and investigate any issues associated with any jobs.

4. Workflows

Each task is created with a specified workflows. iSkeddy comes with number of workflows out of the box. Workflow is the definition, execution and automation of processes. It can be running a program, sending a email, transfer a file to ftp etc.

In here I’m going to talk about some of the most commonly used workflows and how to use them when defining task.

4.1 RUN_PROGRAM

There are some common tasks which needs to be executed at scheduled intervals. If you want Windows to work properly and always strive for a faster system performance, you will have to run backups, use disk cleanup,disk defragmentation or some custom maintenance scripts.

Running these tasks can be really painful at times, because you have to do it manually over and over again. It’s far better if you can automate them using iSkeddy.

Using RUN_PROGRAM workflow you set to run any program on a specified schedule. Here are the available parameters

Image 4.1.1 RUN_PROGRAM parameters

RUN_PROGRAM

You can get to the above window by going to ‘Task Editor’ window -> ‘Edit’ next to Parameters option.

The available parameters are

  • Domain – Identifies the domain to use when starting the process.
  • Password – The user password to use when starting the process. Password will be encrypted.
  • UserName – The user name to be used when starting the process.
  • ProgramName – Absolute path of the program.
  • CommandArgs – Optional command arguments.
  • Silent – True, to ignore the error reported on the stderr. Otherwise false. (Default: True).
  • Timeout – Wait time for this task to complete (Default: -1).
  • UseShellExecute – Indicating whether to use the operating system shell to start the process.
  • WaitForExit – Wait for application to exit (Default: True).
  • WorkingDirectory – When the UseShellExecute property is false, gets or sets the working directory for the process to be started. When UseShellExecute is true, gets or sets the directory that contains the process to be started.

4.2 FILE_WATCH

This task monitors the specified file for changes.

Image 4.2.1 FILE_WATCH parameters

WFParamsEditor

Here are the list of parameters supported by this workflow

  • File Path – Path of the file to watch.
  • Delay – Start the file watch after specified time interval (Default: 0)
  • Timeout – Indicate the time waits for the file watch complete (Default: -1).
  • Watch If File Exists – Watch even if file exists (Default: false).

Other built-in workflows will be detailed out shortly. In the mean time, please go to http://www.iskeddy.com to download and try it for free.

 

Cinchoo – Using Array/Collections in Configuration Object

How to use Array or Collections in configuration object

Download sample source files

Cinchoo – Application configuration is one of the feature Cinchoo framework provides for the developer.  Application configuration is the information that application reads/writes at run time from the source. .NET already provides extensive support through several predefined configuration section handlers for the developers. But require some pluming development work on top of them to complete the exercise. Cinchoo framework simply them with less code and read/write configuration values to the underlying data sources seamlessly.

In this section, I’m going to detailed out on using Array or Collection objects in your configuration object. ChoXmlSerializerConfigurationSection is suitable for this type of needs. It is a section used to consume configuration object containing user defined objects. In here, we going to explore a way to consume array of objects from configuration source.

Listing 1.1 Configuration Object

[ChoXmlSerializerConfigurationSection("appSettings")]
public class AppSettings : ChoConfigurableObject
{
    [ChoPropertyInfo("name", DefaultValue = "Raj")]
    public string Name;

    [ChoPropertyInfo("employees")]
    [XmlArray("employees")]
    public ObservableCollectionEx<Employee> Employees = new ObservableCollectionEx<Employee>();

    protected override bool OnBeforeConfigurationObjectLoaded()
    {
        if (Employees != null)
            Employees.Clear();

        return base.OnBeforeConfigurationObjectLoaded();
    }

    protected override void OnAfterConfigurationObjectLoaded()
    {
        if (Employees != null)
            Employees.WireCollectionChangedEvent();

        Console.WriteLine(ToString());
        base.OnAfterConfigurationObjectLoaded();
    }
}

The code above illustrates about defining configuration object. It contains two members, name and Employees. name is a simple string configuration member. Cinchoo does the job of consuming it from source out of the box. The next member, Employees, is the collection of Employee objects. In order to consume from the source, it can be defined as List<T>, Array or ObservableCollection. But the problem here is the receiving change notification when collection changed or when one of its item gets changed. That’s why I’ve created a modified version of ObservableCollection class, to meet the purpose of two way binding. Take a look at the attached sample for its implementation. Besides that you must decorate this member with XmlArrayAttribute.

Next you need to take care few house keeping tasks of managing this collection member. Override OnBeforeConfigurationObjectLoaded method, to clear the collection before the configuration object is loaded from the source. On to next, override OnAfterConfigurationObjectLoaded method to wire the collection changed event to property changed event. I know it is kind of confusing. Read the sample to get clarity on it.

Now let’s define and consume the object as shown below

Listing 1.2 Main Method

static void Main(string[] args)
{
    AppSettings appSettings = new AppSettings();

    int random = Math.Abs(ChoRandom.NextRandom());
    Employee e1 = new Employee() { Id = random, Name = "Name{0}".FormatString(random) };
    appSettings.Employees.Add(e1);

    ChoConsole.PauseLine();

    foreach (Employee o in appSettings.Employees)
        o.Name = o.Name + "1";

    ChoConsole.PauseLine();
}

Build and run the sample, the configuration file looks as expected below

Listing 1.3 Configuration Output

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="appSettings" type="Cinchoo.Core.Configuration.ChoXmlSerializerSectionHandler, Cinchoo.Core" />
  </configSections>
  <appSettings>
    <Name>Raj</Name>
    <employees>
      <Employee Id="1416154546" Name="Name141615454611" />
      <Employee Id="1121887979" Name="Name11218879791" />
    </employees>
  </appSettings>
</configuration>

That’s all, the object is ready to be used to consume collections. Download and try for yourself. Thanks.

Cinchoo – Consume JSON format configuration values

1. Introduction

Cinchoo is an application framework for .NET. One of the main functionality it provides to users is the application configuration management. Application configuration is information that an application reads and/or writes at run-time from the source.

Please visit jump start article [Cinchoo – Simplified Configuration Manager] for more information about Cinchoo configuration manager.

In this section, I’m going to show you how to consume JSON data as configuration values using the Cinchoo configuration framework. It gives an opportunity to use industry standard message format as configuration source.

Let’s start this exercise using the below values defined in test.json file. This file can be places in any folder. You must specify the absolute path of this file while defining configuration object. Please note, if the file is not present, it will be created automatically the very first time the application starts.

Listing 1.1 Sample JSON file (test.json)

{"name":"Mark","address":"10 River Road, 08837","RunDate":"8\/16\/2015 6:16:36 PM"}

2. How to use

  • Open VS.NET 2010 or higher
  • Create a sample VS.NET (.NET Framework 4) Console Application project.
  • Download the latest Cinchoo.Core.Configuration.JSON via nuget. (Nuget Command: Install-Package Cinchoo.Core.Configuration.JSON)
  • Use the Cinchoo.Core.Configuration namespace
  • Define a configuration object ‘ApplicationSettings‘ as below.

Listing 2.1 Defining Configuration Object

[ChoJSONConfigurationSection("appSettings", ConfigFilePath = "test.json")]
public class ApplicationSettings : ChoConfigurableObject
{
    [ChoPropertyInfo("name", DefaultValue = "Mark")]
    [ChoNotNullOrWhiteSpaceValidator]
    public string Name
    {
        get;
        set;
    }

    [ChoPropertyInfo("address", DefaultValue = "10 River Road, 08837")]
    public string Address;

    [ChoPropertyInfo("Date", DefaultValue = "%%NOW^MM/dd/yyyy HH:mm:ss%%")]
    public DateTime RunDate;

    protected override void OnAfterConfigurationObjectLoaded()
    {
        Console.WriteLine(ToString());
    }
}

The code above illustrates about defining configuration object. First thing defines a configuration object (ex. ApplicationSettings) class from ChoConfigurableObject, it indicates that this object is a configuration object. And it must be decorated with ChoJSONConfigurationSectionAttribute to complete the definition. This attribute enables the configuration object to read and write configuration values from ‘test.json’ file.

Define three members Name, Address and RunDate as public fields or properties with get and set in it. Decorate them with ChoPropertyInfo attribute to indicate that they are configuration members. In this sample, Name property is given name as property name with defaultvalue as ‘Mark‘. It means that this member will be defaulted to default value when the value is missing in the configuration file. Respectively declare other members for each appSettings key. It is a very simple and clear way of specifying configuration members using attributes.

Once you have class declared as above, it is now ready to consume ‘test.json’ values as simple as creating object out of it. Any changes made to this object and/or in the test.json file will be exchanged automatically. The sample code is as below:

Listing 2.2 Main method

static void Main(string[] args)
{
    ApplicationSettings applicationSettings = new ApplicationSettings();
    Console.WriteLine(applicationSettings.Name);
    ChoConsole.PauseLine();
}

We start by creating a new instance of ApplicationSettings object. That’s all. All the heavy lifting of loading appSettings values to the object is done by the Cinchoo framework under the hood.

Just compile and run the above sample, it will output as below:

Listing 2.3 Output of Test1.exe

-- Cinchoo.Core.Configuration.JSON.Test.AppSettings State --
        Name: Mark
        Address: 10 River Road, 08837
        RubDate: 8/17/2015 10:02:08 PM

Press ENTER key to continue...

Now let’s see how the application picks up the changes made to the test.json file reflected in the output. Open test.json file, edit the name from ‘Mark‘ to ‘Tom‘ and save the file.

Listing 2.4 Output of Test1.exe when changing the values in the App.Config file

-- Cinchoo.Core.Configuration.JSON.Test.AppSettings State --
        Name: Mark
        Address: 10 River Road, 08837
        RubDate: 8/17/2015 10:02:08 PM

Press ENTER key to continue...
-- ChoStandardAppSettingsConfig.Test.ApplicationSettings State --
        Name: Tom
        Address: 10 River Road, 08837
        RubDate: 8/17/2015 10:02:08 PM

That’s all. A sample project is attached. Download and try for yourself.