How do I hack makefiles?
If you have built from source, then you know the relief when nothing interesting comes out of ./configure && make && make install. In fact, the less interesting the output of these commands, the better.
But occasionally, the source build process is so horrifying that you end up having to modify the configure script or makefile yourself.
So far I have only been able to do this when I was lucky enough to find some poor, destitute stranger who had pretty much the same problem as me ( most recent I can think of is GNUTLS, where I had to adjust the version requirements for nettle ) and that is a problem -- there must be some way to learn this myself.
Is this just something that comes with time and experience, or does anyone have a reliable guide or resource for modifying makefiles and configure scripts? I would appreciate advice / discussion: I am tired of "getting lucky" with these!
The GNU build system is a huge mess. This comment is an introduction to that mess. It might help you. It's mostly a rant.
Most
./configurescripts aren't written by hand. That's why they look like casseroles. In fact, most Makefiles also aren't written by hand. We'll get to that.Make is nothing more or less than a language for describing dependencies for files; for building files into others. This is all very good. If you want to use Make in a sensible way, by expressing e.g. "build object files from these
.cfiles, then link them into this executable", you can read the GNUmakemanual. Read sections 2 and 4, skim section 6, and keep the rest as reference. Sucklesssbase,musl, and the Tiny C Compiler all have excellent Makefiles.Here comes the mess.
Autotools
Back in the day, software had to deal with a bunch of very incompatible compliation environments. Sometimes functions had different signatures, sometimes they weren't defined. So people started writing shell scripts to explore the computer (libraries, headers, etc.), and generate the Makefile automatically from a template (
Makefile.in).But configure scripts are tedious to write by hand. So GNU Autoconf was born. Autoconf takes
configure.acscripts and makesconfigurescripts… which make Makefiles… which make programs. And as a bonus, Autoconf scripts are written in an ancient language, M4, which you will never see anywhere else.Unfortunately, Make isn't smart enough to keep track of which files depend on each other. If you do it manually but incorrectly, the program might not build, or worse, might build with outdated parts. Fear not! There is a tool called GNU Automake, which explores your source tree and generates
Makefile.in(which configure uses to make the real Makefile) automatically! Except Automake can't determine everything about your project, so you need to give it a fileMakefile.amas input.Wouldn't it be great if you didn't have to write Autoconf scripts? Surely someone has a list of functions that don't exist on some computers or whatever. Good news! Autoconf includes
autoscan, which does that for you!To recap:
autoscan.Complexity
Obviously this is all
a load of garbagevery complex. It's ironic that, towards the goal of a free Unix, GNU sacrificed the Unix philosophy to portability. These programs are all subtly interconnected and difficult to reason about. The portability concerns are outdated too! There's no reason you couldn't write these tools as, say, libraries in shell script; or as programs in GNU Make, which, by the way, is incompatible with other Makes.The upshot is that, incredibly, knowing Make will often not help very much when you need to fix builds that use Make. But do write Makefiles for your own projects where possible (i.e. when simple enough). Make is a great tool.
Other projects saw this problem and fixed it in different ways. Let's explore.
Alternatives
Ninja is a replacement for what Make became: a description of file dependencies and commands which are generated by other tools. Ninja is very fast and simpler than Make. Ninja is in charge of executing builds.
CMake is a tool for describing software requirements and finding them on the computer; and for determining which source files depend on which others. It is basically a replacement for configure scripts and what's involved in generating them. It's clunky, but better than casserole. CMake can generate Makefiles, Ninja files, and files for other build systems. CMake is in charge of planning builds.
Meson is like CMake, except the configuration language is quite different. As far as I can tell, the project emphasizes automatic configuration (edit the config specification as necessary) over CMake's occasional manual tweaks (edit the generated variables as necessary). It's hard to recommend one of these over the other, although Meson can use CMake dependencies and its syntax is more C-like. Meson also generates Ninja files.
SCons and waf replace the entire build system, from configuration to build execution. I can't talk much about these since I don't use them.
Hopefully this was useful for someone. My relationship with GNU builds became very strained recently. May yours remain cordial.
Edit.
s/a load of garbage/very complex/, don't be rude.One note: GNU is Not Unix. It's based off of it, but is not meant to be compatible with it. Personally I value the Unix philosophy---or rather the Unix prophecy or the Unix aphorism, to be more accurate---, it is useful at times, but not all the time, and not often, even. Portability was vital for free software, and it was one of the pillars that helped make it what it is today. And what auto{tools,make,...} do is not really possible that way. Cf. the alternatives cited, none of them adhere to such proverb.
@mydriasis: The problem you're having stems from the fact that you seem to not know the distinction between
make(the language for build recipes) andautotools automake ...suite of programs and their output (the "GNU Build System"). That's normal because the use of the latter is so widespread in certain spheres. Just see if it has aconfigure.ac, then you're most probably dealing with an autotools based project. It probably will have anautogen.shscript too. You generally edit one or more of the*.acand**/*.amfiles and run./autogen.sh all.m4is a macro language like the C preprocessor, but general purpose. The ac and am files are shell scripts or makefiles respectively intermingled with macro expansions that import shell and make fragments. Autotools is nothing but a library of these macros and related stuff. See: Autotools / GNU Build System docs and examples, and maybe this which is an alternative guide to autotools.makeis a program used by this system. It's fairly easy to suffice with makefiles in projects that do not need to be very portable, many do that and that's advisable, but autotools has its purpose and is not the pile of garbage people like to make it seem to be. Portability is not limited to binary portability. There are hundreds of Linux distros with varying sets of tools and a rather wide selection of C standard libraries, and many other Unix-like systems like BSDs and Macs etc and many versions, customised or not, of these, plus those of the other libraries the program might use; and then there are operating systems that are not Unix-based, like Windows. It's an incredibly complex task to be portable to all or even most of these, and be able to tell the user (and the developer who'll write the docs) which sets of libraries are required to build a valid library or executable. Even if you stick strictly to POSIX standards, there is no guarantee these herd of OSes are compliant. It's the complexity of the task at hand that renders the tool complex.make, among all the alternatives and things, is the most widely available tool to write your build recipes in. If you don't use it or something that generates a Makefile, you'll require most of your users to install yet another tool just to build your source tree, or they'll have to have a binary package available somewhere. Autotools is widely used, is backed by GNU, and uses standard Unix programs or GNU versions thereof to work (shell scripts, makefiles, m4 macros). IMHO the only better alternative is pure make, which is used successfully by many projects like all the BSDs and the Linux kernel (the kernel makefile is pure artwork, very well designed).edit: emphasis
Typical kids, criticising things That Work and are historically essential, right? :P
I'll second the Linux Makefile. It's beautiful and well-documented.
Thank you so much!
You're welcome!
That's not true at all. You will encounter m4 if you ever get to experience the joy of configuring sendmail. In which case m4 is actually simpler than the native configuration script.
I've been wracking my brain and in a couple decades of IT work I've only ever seen it in sendmail and makefiles. On a blind bet I'd wager it was used for usenet news services like a/b/c but I never had the pleasure. Funny how something as obtuse as m4 was at one time considered a beneficial simplification of older, more arcane and evil methods.
M4 doesn't seem that strange to me; it's just a macro language for processing text. There are other languages meant for the same thing, much like awk and sed. They just aren't commonly used any more.
I don’t understand the purpose of an object file. Is it like an intermediary pickle file for the compiler?
AFAIK an object file is like a module, and can expose or hide functionality a la OOP classes. They can also be composed and linked in a varied manner to create a/some library/-ies and/or program/s from the same code base.
What’s the purpose of saving one of these files? Machine code isn’t legible, is there any reason to save an object file unless you’re developing and testing a linker?
If you recompile the project but only change one source file, the other existing object files can be reused, which speeds up the build.
You can also mix object files produced in different ways. For instance, a C compiler, a C++ compiler, and an assembler can all make object files that are usable together.
Object files separate the concern of compilation (e.g. language parsing, optimization, instruction selection, register allocation) from that of linking (i.e. executable layout).
Imagine you're a compiler. You're reading plain text and producing an executable file.
When you compile a function, you emit machine code (add this to that, check if it's zero, otherwise jump backwards an instruction). Sometimes, you need to reference code or a memory location that hasn't been defined. This is done by allocating space for that instruction, but leaving the operand out (jump to ____; multiply ____ by two).
Why do you need to reference undefined stuff? Part of your job is determining the memory layout that the machine code will have when the program is running. But if you haven't compiled that other function yet, you don't know where it will live in memory. And you can't really guess, because you don't know how many bytes each compiled function uses until after it's compiled.
You might also reference code from a dynamic library, which is code that, by definition, has no assigned memory location until the program starts. The location is unknowable at compile time.
There is a logical point like this for many compiled languages. The strategy is to leave blanks in the machine code, then keep track of those blanks. Blank spots have names called symbols. A file containing blanks, symbols, and machine code is an object file.
If the job of a compiler is to produce machine code, then the job is finished once all internal references are resolved and the blanks are filled (mostly). Now you have a blob of binary that is independent of language-specific details (mostly). When you want to finish up by merging object files, possibly produced in wildly different ways (with different languages or compilers), you can use a separate program called a linker that doesn't need to know anything about programming languages.
Heroic, amazing!
I would guess that the skills required to hack Makefiles are the same as those required to write them in the first place; and there's a lot of resources when it comes to that.
As a related topic: I am quite glad to see
mesonandninjasupplantingmakein many newer projects. For me, these tools are a lot more managable to deal with.Yeah, I was going to suggest the GNU Make manual, but I don't know how helpful that actually is. There are plenty of intro level tutorials out there that may be useful as well, but I find that if I get in trouble with a makefile it's almost always something really weird that I'll either have to find the right SO question or end up reading the manual.
RE: Newer build systems, I'll put in a word for SCons -- the best feature being that it's built in python, so you can literally jump into a PDB debug shell anywhere your build starts running into issues. The documentation can be a bit spotty if you go too far beyond the basics (e.g., adding new language targets, etc.) though.