James has a blog

I spent a while trying to debug a runc problem where it would always get an EACCES error writing the cpu.cfs_period_us file in a cpu cgroup.

The problem turned out to be that I had not enabled CONFIG_CFS_BANDWIDTH in my kernel build. Presumably, when runc tries to write the file, it passes O_CREAT and cgroupfs doesn’t let it create a new file, which leads to the somewhat surprising error.

So, if you get this error, just turn on CONFIG_CFS_BANDWIDTH :)

This is probably something lots of other people have tried and burnt themselves with, but anyway, this time it’s my turn.

The goal is, given an arbitrary protobuf, can we write an API that applies default values to it? Normally we would create a prototype protobuf object with the defaults and merge our current object into it, updating the prototype object with current values. However, in Go, this would erase the type information and mean we would have to do some ugly casting (it’s easy to avoid this in C++ by using templates).

So me, being clever, comes up with this construct (in Go):

message := proto.Message{} // The message to update
prototype := SomeStruct{} // The defaults to apply

// Merge the message into the prototype, which updates any
// default values with values from the message, and retains
// any that the message doesn't specify.
proto.Merge(prototype, message)

// Now here's the trick. We update the message with the updated
// prototype. The prototype is (potentially) a superset of the
// message since it has filled in missing default values, but
// stil has the same values it copied from the message.
proto.Merge(message, prototype)

The reason this doesn’t work is that merging repeated fields concatenates them. So if the original message had any repeated fields, the double merge just doubled them.

And that’s how I spent the afternoon debugging.

I just spent a few hours trying to get systemd to spawn a getty on the vt console for a Linux image that I was building with Buildroot. It turns out that it helps to read the right documentation, which in this case was a blog about systemd console handling. The money quote for me was:

In systemd, two template units are responsible for bringing up a
login prompt on text consoles:

getty@.service is responsible for virtual terminal (VT) login
prompts, i.e. those on your VGA screen as exposed in /dev/tty1
and similar devices.

serial-getty@.service is responsible for all other terminals,
including serial ports such as /dev/ttyS0. It differs in a couple
of ways from getty@.service: among other things the $TERM environment
variable is set to vt102 (hopefully a good default for most serial
terminals) rather than linux (which is the right choice for VTs
only), and a special logic that clears the VT scrollback buffer
(and only work on VTs) is skipped.

My Buildroot config wired up a serial-getty but not a getty. Once I added the getty@tty1.server, systemd did what I wanted. Next step is to understand how to control this in Buildroot …

Here’s the smallest amount of code I could come up with to set the user agent when making a HTTP request in Go:

type UserAgent string

func (u UserAgent) RoundTrip(r *http.Request) (*http.Response, error) {
    r.Header.Set("User-Agent", string(u))
    return http.DefaultTransport.RoundTrip(r)
}

http.DefaultClient.Transport = UserAgent("my-great-program")

Bazel defaults to building code in a sandbox that remounts most of the filesystem read-only. This means that if you are using ccache (Fedora, for example, will enable it by creating appropriate symlinks when you install the package) the compile job will fail because it can’t write to the cache directory.

The straightforward fix to this is to create a bazelrc file which specifies the Bazel sandbox_writable_path flag to make the cache directory writeable.

$ cat ~/.bazelrc
build --sandbox_writable_path=/home/jpeach/.ccache

Is ccache actually worth it with Bazel? Good question!

In this post, I remember how to use the alternatives(8) mechanism to make clang’s lld linker the default.

First, tell alternatives that lld is available and set it at a high priority:

$ sudo alternatives --install /usr/bin/ld ld /usr/bin/lld 80
$ sudo alternatives --auto ld

The, just verify that it worked:

$ alternatives --display ld
ld - status is auto.
 link currently points to /usr/bin/lld
/usr/bin/ld.bfd - priority 50
/usr/bin/ld.gold - priority 30
/usr/bin/lld - priority 80
Current `best' version is /usr/bin/lld.
$ ld --version
LLD 6.0.1 (compatible with GNU linkers)

The canonical way to feed your project to cquery is to generate a compile_commands.json file, possibly using cmake. Every time I switch my Mesos work to the cmake build, I live to regret it, either because the component I’m working on isn’t implemented in the cmake build or I end up wanting to install the build (and that isn’t implemented in cmake).

So here’s a .cquery file that makes cquery work pretty well with Mesos:

%clang
%cpp -std=c++11
-Iinclude
-Isrc
-Ibuild/src
-Ibuild/include
-I3rdparty/stout/include
-I3rdparty/libprocess/include
-Ibuild/3rdparty/googletest-release-1.8.0/googletest
-Ibuild/3rdparty/glog-0.3.3/src
-Ibuild/3rdparty/picojson-1.3.0
-DLIBDIR="/lib"
-DVERSION="1.0"

A while ago I tried cquery with Vim, but was a bit unsatisfied with the integration. I could probably have altered my keybindings to perform LSP searches instead of cscope searches, but I also really quite like the integrated tags stack you get with the built-in Vim cscope support.

So I thought that it couldn’t be too hard to implement the cscope line protocol with a cquery backend, and it turns out that it wasn’t. I wrote cscope-lsp on and off during the week and it’s almost at a useful point. Most search types work, and vim will drive it without too much configuration. The main missing feature is that I need to resolve a LSP Location to an actual text string for the cscope results to be meaningful, but in many cases, cscope-lsp just jumps directly to the result you want, which is a big improvement for my workflow.

Writing this down quickly before I forget.

When debugging a std::string from GNU libstdc++, the debugger typically won’t show you the actual representation.

First, you need to turn off the pretty printer (assuming that it worked in the first place):

(gdb) p reregisterSlaveMessage.resource_version_uuid_.ptr_
$13 = (std::string *) 0x7f4d98d5e970
(gdb) p *reregisterSlaveMessage.resource_version_uuid_.ptr_
$14 = "\022\020K|\n\225\064\246CE\222\350\275\315t", <incomplete sequence>
(gdb) disable pretty-printer
2 printers disabled
0 of 2 printers enabled
(gdb) p *reregisterSlaveMessage.resource_version_uuid_.ptr_
$15 = {
  static npos = <optimized out>,
  _M_dataplus = {
    <:allocator>> = {
      <:new_allocator>> = {<no data fields>}, <no data fields>},
    members of std::basic_string<char std::char_traits>, std::allocator<char> >::_Alloc_hider:
    _M_p = 0x7f4d98d67068 "\022\020K|\n\225\064\246CE\222\350\275\315t", <incomplete sequence>
  }
}

Next, you need to know that the internal structure of std::string is prepended to the actual string data, so you need to cast and subtract from the data pointer to find the length and refcount. In the example below, I show the std::string::_Rep structure that is laid out inn memory immediately before the _M_p string data pointer:

(gdb) p (((std::string::_Rep *)0x7f4d98d67068) - 1)[0]
$16 = {
  <:basic_string std::char_traits>, std::allocator<char> >::_Rep_base> = {
    _M_length = 18,
    _M_capacity = 18,
    _M_refcount = -1
  },
  members of std::basic_string<char std::char_traits>, std::allocator<char> >::_Rep:
  static _S_max_size = <optimized out>,
  static _S_terminal = <optimized out>,
  static _S_empty_rep_storage = <optimized out>
}

Recently, it occurred to me that running a containerized task is concentually very similar to having a remote session on an anonymous compute agent. The traditional way for operators to influence (i.e. configure, control, log) the environment of a remote user session is by the use of PAM modules. One of the applications that I had in mind was the use of the pam_loginuid module to set the linux audit ID so that containers audit events can be attributed to the task user rather than to the container orchestrator. In describing this idea to others, I was a little surprised to find that PAM was quite unfamiliar to a lot of people. Even though it is such a fundamental part of every UNIX-like system, it seems to be quite an esoteric subsystem these days.

Anyway, I wrote a design document describing how I think this would work with the Mesos containerizer and we discussed it in the Mesos Containerization Working Group which was recorded on YouTube. The Mesos integration seems a little tricky, but quite do-able, and I think this would have some fun applications. Hopefully I will get time to actually implement it :)