I published my generative art side project last week, when it was about 90% done and 50% documented. So I’ll be trying to get those numbers up, mostly by adding more examples to https://bauble.studio/help/. If time allows, I need to write a blog post about the project and record a demo, but I think that’s going to be a next week project…
On today’s stream I’ll talk about yesterday’s outage (I already posted one small update today) and write some code. There’s an issue with previews and rss feeds, and some small PRs. I’ll pick pagination back up if the bugs leave time.
I’ve gotten Recheck to alpha testing and that caught plenty of bad data in the Lobsters database. I expected 10-20 records but got 308, which was a bit of a surprise. I need to improve the output a bit for it, but I’d like to demo this on the Thursday stream. And then more broadly this week, I’m trying to get it to the point I can onboard a couple people to a closed beta.
Starting to learn Akkadian, using Worthington’s Complete Babylonian initially.
This is probably the first dedicated studying I’ve done since leaving university; everything I’ve learned since has been through practical hands-on means like programming projects. This is also the first foreign language I’m actually seriously attempting to learn, as the terrible UK(?) method of teaching languages (let’s memorise verb tables!) burned me out when I was 16.
Last year I started writing an IDL / data encoding format for zero-copy IPC. The first implementation was in Rust and I planned to open-source it, but in May that plan got blocked by Rust’s lack of dynamically-sized thin pointers (https://john-millikin.com/rust-and-dynamically-sized-thin-pointers). My options were to release a stable API and accept the overhead of thick pointers forever, or release a thin-pointer API that can compile on stable but relies on pointer-punning tricks that MIRI doesn’t like.
So now I’m writing a new reference implementation in Go, and have also started writing up a proper spec + language-independent test suite so that I can verify correctness without having to do a line-by-line comparison with the Rust version.
Yesterday I uploaded the first bits to GitHub, a very rough draft of the spec (not even a README yet): https://github.com/jmillikin/idol
The basic idea is you start with a Protobuf-style IDL file like this:
The IPC layer is separate and could be anything, since I’d like this to work for AF_UNIX, shared memory, D-Bus style routing proxies, and whatever else that two processes on the same machine might want to transmit structured data.
There’s also a Protobuf-style compact wire format that works well for RPC (for example with gRPC), but documenting and publishing that will probably need to wait for a bit due to impending time constraints (Factorio 2).
Sounds interesting! What’s the design goals/decisions that make Idol different from Cap’n’Proto?
Also I’d encourage you to post the Rust implementation publicly. It’ll greatly lower the bar if someone else wants to implement it in Rust at some point.
And maybe even make a first non zero-cost version. I understand you have strong feelings about the performance but I think having something that works is better than not, especially for a young project where the code already exists.
Obviously it’s your call, just my 2cts :)
I’m also curious about how Go deals with DSTs. I haven’t looked it up, but do you know a good resource to learn about it?
I assume in the code it means dealing with unsafe.Pointer.
What’s the design goals/decisions that make Idol different from Cap’n’Proto?
A full comparison with Cap’n’Proto would be too long for a comment, but in summary:
Idol is intended to be simple to understand and implement. The spec is verbose because I’m bad at technical writing, but the goal is to have something like JSON where a person of ordinary skill in the art could just sit down with an empty directory and have a “hello world” decoder within a few hours.
Cap’n’Proto is somewhat infamously complex, to the point where its official documentation on the encoding says “Field offsets are computed by the Cap’n Proto compiler. The precise algorithm is too complicated to describe here, but you need not implement it yourself,”. Third-party implementations tend to be of the support library, but not the Cap’n’Proto schema compiler.
The Idol binary format supports in-place decoding, where the decoder actually walks the tree and mutates it so that offset calculation and content validation happens at the system boundary. Cap’n’Proto is designed for deferred validation, where receiving a message is “free” but each field access is failable.
The Idol reference library includes a deferred-validation module, but it’s intended more for debug tools or special cases like read-only shared memory.
Idol intentionally does not include an IPC/RPC layer – it’s just an IDL and data encoding format. The Cap’n’Proto RPC protocol continues to be a source of continued development velocity within that project, and I’m hoping for Idol to be more of a JSON-style “frozen” format.
The reference implementation might see small improvements for performance or correctness, but it’ll never have (for example) an I/O subsystem.
Support for handles (capability-passing) is based on FIDL, where the decoder provides a list of u32 handle IDs and the decoder will inject them into the messages. This allows Idol to be agnostic to the format and size of capabilities – for example they might be file descriptors passed over an AF_UNIX socket, or TLS certificates passed via gRPC headers.
Idol’s simplicity has the cost of limited expressiveness. Its type system is similar to C, and it doesn’t have features such as type parameters or key-value maps because they would make it more difficult to reason about the mapping between IDL and the encoding. In general it’s trying to be a replacement for AF_NETLINK and other schemes based on directly peeking into C struct fields.
Also I’d encourage you to post the Rust implementation publicly. It’ll greatly lower the bar if someone else wants to implement it in Rust at some point.
I’ll post it at some point, but I need to rework it because its public API is based on memory-access patterns that aren’t allowed by MIRI. This will probably mean messages become fat pointers, which means I need to do some research to figure out how to maintain the performance characteristics of thin pointers. Probably the internal code is going to use raw pointers in a lot more places.
Probably the parser will be the first part to be posted, since it’s less dependent on the rest of the API. Rust’s module visibility is helpful here – the Go version sort of has all its innards exposed, so I can’t publish the public API incrementally.
And maybe even make a first non zero-cost version. I understand you have strong feelings about the performance but I think having something that works is better than not, especially for a young project where the code already exists. Obviously it’s your call, just my 2cts :)
People who are interested in a non-zero-cost data encoding would probably just use Protobuf instead.
It’s not exactly a young project (a year old at this point), it’s just that open-sourcing it has taken a more winding road than expected.
I’m also curious about how Go deals with DSTs. I haven’t looked it up, but do you know a good resource to learn about it?
Go basically has two DSTs available, []uint8 for mutable buffers and string for immutable buffers. So all of the types that provide a view into some immutable memory region are wrapping string. My mind has a difficult time reconciling the Go type with my (and Rust’s) definition of “string”, which implies some sort of human-readable text content.
I assume in the code it means dealing with unsafe.Pointer.
Yeah, for wrapping string into other types it has code like this:
All seem like good reasons. I especially like the idea of keeping it focused on the IDL, and upfront validation.
I don’t have any first hand experience with Cap’n’Proto was asking because it’s the main player in that space AFAIK.
Maybe you can copy paste this bit into the README :)
open-sourcing it has taken a more winding road than expected
Yeah I feel you, the 90% rule hits hard.
My mind has a difficult time reconciling the Go type with my (and Rust’s) definition of “string”, which implies some sort of human-readable text content
I agree it’s surprising. I knew that you can cast any byte array to a string but it never crossed my mind to use it for random bytes.
House renovations: Tearing down old plaster in the to-be bar/library in the basement. Tearing down old kitchen. Finishing DIY CNC machine intended to be used for kitchen cabinets.
The eternal project: Accessibility work. Writeup finished, most things for no-vision finished, first run configuration and more wild experiments (seeing-with-sound, positional audio, cursor based raytracing, …) remaining.
Exercise: Tired of sweating in VR fish bowl Beat Saber, whipping up a dance platform and going back to stepmania.
Planning: sourcing parts for a Departed Reality style simulation platform.
Post-offsite depression, it was great and reconnected with everyone after mostly working remote since my daughter was born. Have no energy for anything.
Congrats on the new kid! They’re fun, but exhausting. I don’t know if she’s your first kid or not but if you don’t already know…don’t expect to get a full night’s sleep for like a year at least. 😅
Finishing up an article I’ve been writing for a looooong time, training myself more in Rust by doing some more side projects, searching for a new job, reorganizing/decluttering the apartment after a long stretch of DIY projects, and gearing myself up for EuroRust in October!
Onboarding with a new client and remembering what being at a computer for a full workday feels like (first time in about seven weeks.) Need to sort out various small issues with computers and technology for family & friends.
Outside of that, mostly intending on catching up on gaming now my rig is accessible again and continuing to read.
working on a few scripts I’ve been using to make my day to day life easier at work. There are 20 or so python scripts that automate my workflow, but it would be nice to put them all together into a nice app, but I hate web programming.
On the side, continuing to learn a bit of GLSL and using zig to scaffold a local shadertoy look-a-like. It’s been really interesting doing the things I used to do in C, with ease, but with zig which I find to be a more convenient language.
Oh, and Don’t Starve Together with a few friends this weekend, it’ll be fun!
I’ve been working myself to the bone the past couple of weeks. I’d like to take it easy this week. Maybe just some recreational coding, maybe just catching up on my reading.
I finally decided to start learning Elixir and trying to make small projects with Phoenix and Liveview. I’ve never worked with a functional programming language before (coming from Node/React land) so I’m sure there are a lot of surprises in my way.
Planning to finish the second book of In Search of Lost Time. Proust is a different energy altogether.
I published my generative art side project last week, when it was about 90% done and 50% documented. So I’ll be trying to get those numbers up, mostly by adding more examples to https://bauble.studio/help/. If time allows, I need to write a blog post about the project and record a demo, but I think that’s going to be a next week project…
Continuing my journey with Gleam and OCaml. Already built some small programs with them, but now diving deeper.
On today’s stream I’ll talk about yesterday’s outage (I already posted one small update today) and write some code. There’s an issue with previews and rss feeds, and some small PRs. I’ll pick pagination back up if the bugs leave time.
I’ve gotten Recheck to alpha testing and that caught plenty of bad data in the Lobsters database. I expected 10-20 records but got 308, which was a bit of a surprise. I need to improve the output a bit for it, but I’d like to demo this on the Thursday stream. And then more broadly this week, I’m trying to get it to the point I can onboard a couple people to a closed beta.
Starting to learn Akkadian, using Worthington’s Complete Babylonian initially.
This is probably the first dedicated studying I’ve done since leaving university; everything I’ve learned since has been through practical hands-on means like programming projects. This is also the first foreign language I’m actually seriously attempting to learn, as the terrible UK(?) method of teaching languages (let’s memorise verb tables!) burned me out when I was 16.
Last year I started writing an IDL / data encoding format for zero-copy IPC. The first implementation was in Rust and I planned to open-source it, but in May that plan got blocked by Rust’s lack of dynamically-sized thin pointers (https://john-millikin.com/rust-and-dynamically-sized-thin-pointers). My options were to release a stable API and accept the overhead of thick pointers forever, or release a thin-pointer API that can compile on stable but relies on pointer-punning tricks that MIRI doesn’t like.
So now I’m writing a new reference implementation in Go, and have also started writing up a proper spec + language-independent test suite so that I can verify correctness without having to do a line-by-line comparison with the Rust version.
Yesterday I uploaded the first bits to GitHub, a very rough draft of the spec (not even a README yet): https://github.com/jmillikin/idol
The basic idea is you start with a Protobuf-style IDL file like this:
And the messages get encoded into a vtable+data format (conceptually similar to FIDL):
The IPC layer is separate and could be anything, since I’d like this to work for
AF_UNIX, shared memory, D-Bus style routing proxies, and whatever else that two processes on the same machine might want to transmit structured data.There’s also a Protobuf-style compact wire format that works well for RPC (for example with gRPC), but documenting and publishing that will probably need to wait for a bit due to impending time constraints (Factorio 2).
Sounds interesting! What’s the design goals/decisions that make Idol different from Cap’n’Proto?
Also I’d encourage you to post the Rust implementation publicly. It’ll greatly lower the bar if someone else wants to implement it in Rust at some point.
And maybe even make a first non zero-cost version. I understand you have strong feelings about the performance but I think having something that works is better than not, especially for a young project where the code already exists. Obviously it’s your call, just my 2cts :)
I’m also curious about how Go deals with DSTs. I haven’t looked it up, but do you know a good resource to learn about it?
I assume in the code it means dealing with
unsafe.Pointer.A full comparison with Cap’n’Proto would be too long for a comment, but in summary:
u32handle IDs and the decoder will inject them into the messages. This allows Idol to be agnostic to the format and size of capabilities – for example they might be file descriptors passed over anAF_UNIXsocket, or TLS certificates passed via gRPC headers.Idol’s simplicity has the cost of limited expressiveness. Its type system is similar to C, and it doesn’t have features such as type parameters or key-value maps because they would make it more difficult to reason about the mapping between IDL and the encoding. In general it’s trying to be a replacement for
AF_NETLINKand other schemes based on directly peeking into Cstructfields.I’ll post it at some point, but I need to rework it because its public API is based on memory-access patterns that aren’t allowed by MIRI. This will probably mean messages become fat pointers, which means I need to do some research to figure out how to maintain the performance characteristics of thin pointers. Probably the internal code is going to use raw pointers in a lot more places.
Probably the parser will be the first part to be posted, since it’s less dependent on the rest of the API. Rust’s module visibility is helpful here – the Go version sort of has all its innards exposed, so I can’t publish the public API incrementally.
People who are interested in a non-zero-cost data encoding would probably just use Protobuf instead.
It’s not exactly a young project (a year old at this point), it’s just that open-sourcing it has taken a more winding road than expected.
Go basically has two DSTs available,
[]uint8for mutable buffers andstringfor immutable buffers. So all of the types that provide a view into some immutable memory region are wrappingstring. My mind has a difficult time reconciling the Go type with my (and Rust’s) definition of “string”, which implies some sort of human-readable text content.Yeah, for wrapping
stringinto other types it has code like this:Thanks for the thorough reply!
All seem like good reasons. I especially like the idea of keeping it focused on the IDL, and upfront validation.
I don’t have any first hand experience with Cap’n’Proto was asking because it’s the main player in that space AFAIK.
Maybe you can copy paste this bit into the README :)
Yeah I feel you, the 90% rule hits hard.
I agree it’s surprising. I knew that you can cast any byte array to a string but it never crossed my mind to use it for random bytes.
House renovations: Tearing down old plaster in the to-be bar/library in the basement. Tearing down old kitchen. Finishing DIY CNC machine intended to be used for kitchen cabinets.
The eternal project: Accessibility work. Writeup finished, most things for no-vision finished, first run configuration and more wild experiments (seeing-with-sound, positional audio, cursor based raytracing, …) remaining.
Exercise: Tired of sweating in VR fish bowl Beat Saber, whipping up a dance platform and going back to stepmania.
Planning: sourcing parts for a Departed Reality style simulation platform.
Post-offsite depression, it was great and reconnected with everyone after mostly working remote since my daughter was born. Have no energy for anything.
Congrats on the new kid! They’re fun, but exhausting. I don’t know if she’s your first kid or not but if you don’t already know…don’t expect to get a full night’s sleep for like a year at least. 😅
First kid yeah, she is almost 8 months old and it’s truly exhausting but very rewarding
Finishing up an article I’ve been writing for a looooong time, training myself more in Rust by doing some more side projects, searching for a new job, reorganizing/decluttering the apartment after a long stretch of DIY projects, and gearing myself up for EuroRust in October!
This week it’s all about work. Not sure I’m going to get to any side projects but if I do I’m going to continue my journeys with Go.
Onboarding with a new client and remembering what being at a computer for a full workday feels like (first time in about seven weeks.) Need to sort out various small issues with computers and technology for family & friends.
Outside of that, mostly intending on catching up on gaming now my rig is accessible again and continuing to read.
working on a few scripts I’ve been using to make my day to day life easier at work. There are 20 or so python scripts that automate my workflow, but it would be nice to put them all together into a nice app, but I hate web programming.
On the side, continuing to learn a bit of GLSL and using zig to scaffold a local shadertoy look-a-like. It’s been really interesting doing the things I used to do in C, with ease, but with zig which I find to be a more convenient language.
Oh, and Don’t Starve Together with a few friends this weekend, it’ll be fun!
Just wrap it with Flask.
Hopefully applying for an NLnet Fediversity grant.
Also going through Hackerrank exercises to prepare for interviews, as I’m looking for a job. Kinda fun when there’s no time pressure.
I’ve been working myself to the bone the past couple of weeks. I’d like to take it easy this week. Maybe just some recreational coding, maybe just catching up on my reading.
I finally decided to start learning Elixir and trying to make small projects with Phoenix and Liveview. I’ve never worked with a functional programming language before (coming from Node/React land) so I’m sure there are a lot of surprises in my way.
Planning to finish the second book of In Search of Lost Time. Proust is a different energy altogether.
Welcome to Elixir! What are you using to learn?
Dipping my toes through the official docs: https://hexdocs.pm/elixir/introduction.html