Hi @thedanfields, thank you for your question!
Can you try using this as the schema you pass to CreateDataFrame (The nested Dictionary corresponds to the MapType and not StructType):

var schema = new StructType(
                new List<StructField> {
                    new StructField("Identifier", new LongType()),
                    new StructField("NestedComplexMap", new MapType(new IntegerType(),
                        new MapType(new StringType(), new LongType()))),
                }
            );

It showed me the following result for PrintSchema and Show:

@Niharikadutta !!! Thanks so much.

Out of curiosity, what c# type should I use for a StructType, if I wanted to keep the original schema I provided?

Thanks again for your time and attention! Really appreciate it.

I'm not sure I understood your question, but if you want to use StructType as the original schema, you would need to change your nestedComplexMap to use StructType instead of Dictionary in the nested type.

Hey @Niharikadutta, thanks for your response.

I see why I was unclear.

Ultimately I do not have control of the schema, it's being dictated by a process upstream. My goal is to be able to use the GenericRow to create some fake data to mock the result of that upstream process.

I am stuck with this schema:

new StructType(
    new List<StructField> {
        new StructField("Identifier", new LongType()),
        new StructField("NestedComplexMap", new MapType(new IntegerType(),
            new StructType(new List<StructField>() {
                new StructField("FirstProp", new LongType()),
                new StructField("SecondProp", new LongType()),
                new StructField("ThirdProp", new LongType()),
            }))),
    }
)

I've tried a number of ways of using the GenericRow to adhere to the schema, but I can't seem to figure out the right way.

This example:

 var row = new GenericRow(new object[] { 
                1,  // Identifier
                new Dictionary<int, GenericRow> { // ?? NestedComplexMap ??
                    { 1, new GenericRow(new object[] {1L, 2L, 3L }) },
                    { 2, new GenericRow(new object[] {1L, 2L, 3L }) }
                }
            });
            var dataFrame = _session.CreateDataFrame(new [] { row }, schema);

            var srows = dataFrame.Collect().ToArray();

Results in an Exception stating: System.Threading.SemaphoreFullException : Adding the specified count to the semaphore would cause it to exceed its maximum count.

I've also naively tried substituting the Dictionary<int, GenericRow> with Dictionary<int, NestedComplexType> where NestedComplexType is some class i've defined which contains the FirstProp, SecondProp and ThirdProp properties. I then receive an error related to Spark not supporting the type NestedComplexType, which makes sense.

So given that schema, do you know how I can fabricate a DataFrame ?

Thanks again for your attention here. I really appreciate the support.

@thedanfields Thanks for explaining your scenario and apologies for the late reply. It is currently not possible to create a DataFrame for the schema you have provided (or nested DataFrames in general) because we do not support GenericRow as an argument as of yet. We can talk about supporting this in the next release though. @imback82 , @suhsteve thoughts?

@Niharikadutta, Can you check if we can make the following work?

 var row = new GenericRow(new object[] { 
                1, 
                new Dictionary<int, GenericRow> {
                    { 1, new GenericRow(new object[] {1L, 2L, 3L }) },
                    { 2, new GenericRow(new object[] {1L, 2L, 3L }) }
                }
            });
var dataFrame = _session.CreateDataFrame(new [] { row }, schema);

@Niharikadutta, thanks again for the attention to this.

Just to follow up, While I wasn't able to directly create the needed data, I managed to do it indirectly with two DataFrames and some joins.

@Niharikadutta, thanks again for the attention to this.

Just to follow up, While I wasn't able to directly create the needed data, I managed to do it indirectly with two DataFrames and some joins.

@thedanfields could you elaborate a little more on how you created it indirectly with two DataFrames? Thanks!