[Buildroot] [PATCH 2/5] fs/custom: generate complete, partition-based device images
Maxime Hadjinlian
maxime.hadjinlian at gmail.com
Sat Jan 4 16:38:24 UTC 2014
Hi Yann,
When doing scripting in Buildroot, do we accept the uses of various
bash-ism or do we want to stick to POSIX shell ?
On Fri, Jan 3, 2014 at 6:19 PM, Yann E. MORIN <yann.morin.1998 at free.fr> wrote:
> From: "Yann E. MORIN" <yann.morin.1998 at free.fr>
>
> Contrary to the existing fs/ schemes, which each generate only a single
> filesystem image for the root filesystem, this new scheme allows the
> user to generate more complex images.
>
> The basis behind this is a .ini-like description of the layout of the
> final target storage:
> - the list of device(s)
> - per-device, the list of partition(s)
> - per-partition, the content
>
> It is possible to create MBR- or GPT-based partitoining schemes. Adding
> new ones should be relatively easy (but would need adequate host tools).
>
> For now, the only content possible for partitions is a filesystem. It
> should be pretty easy to add new types (eg. un-formated, or raw blob).
>
> Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding
> more will be relatively easy, provided we have the necessary host
> packages to deal with those filesystems.
>
> The existing Buildroot filesystem generators are re-used as much as
> possible when it makes sense; when it does not (eg. for vfat), a specific
> generator is used.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998 at free.fr>
> Cc: Arnout Vandecappelle <arnout at mind.be>
> Cc: Ryan Barnett <rjbarnet at rockwellcollins.com>
> Cc: Thomas Petazzoni <thomas.petazzoni at free-electrons.com>
> Cc: "Jérôme Oufella" <jerome.oufella at savoirfairelinux.com>
> ---
> docs/manual/appendix.txt | 1 +
> docs/manual/customize-filesystems.txt | 36 ++++
> docs/manual/customize.txt | 2 +
> docs/manual/partition-layout.txt | 298 +++++++++++++++++++++++++++++
> fs/Config.in | 1 +
> fs/custom/Config.in | 16 ++
> fs/custom/boot/gpt | 129 +++++++++++++
> fs/custom/boot/mbr | 64 +++++++
> fs/custom/boot/pre-post | 14 ++
> fs/custom/custom.mk | 25 +++
> fs/custom/fs/ext | 26 +++
> fs/custom/fs/pre-post | 72 +++++++
> fs/custom/fs/vfat | 21 +++
> fs/custom/genimages | 342 ++++++++++++++++++++++++++++++++++
> 14 files changed, 1047 insertions(+)
> create mode 100644 docs/manual/customize-filesystems.txt
> create mode 100644 docs/manual/partition-layout.txt
> create mode 100644 fs/custom/Config.in
> create mode 100644 fs/custom/boot/gpt
> create mode 100644 fs/custom/boot/mbr
> create mode 100644 fs/custom/boot/pre-post
> create mode 100644 fs/custom/custom.mk
> create mode 100644 fs/custom/fs/ext
> create mode 100644 fs/custom/fs/pre-post
> create mode 100644 fs/custom/fs/vfat
> create mode 100755 fs/custom/genimages
>
> diff --git a/docs/manual/appendix.txt b/docs/manual/appendix.txt
> index 74ee8fd..53f4205 100644
> --- a/docs/manual/appendix.txt
> +++ b/docs/manual/appendix.txt
> @@ -6,6 +6,7 @@ Appendix
>
> include::makedev-syntax.txt[]
> include::makeusers-syntax.txt[]
> +include::partition-layout.txt[]
>
>
> // Automatically generated lists:
> diff --git a/docs/manual/customize-filesystems.txt b/docs/manual/customize-filesystems.txt
> new file mode 100644
> index 0000000..fd65c97
> --- /dev/null
> +++ b/docs/manual/customize-filesystems.txt
> @@ -0,0 +1,36 @@
> +// -*- mode:doc; -*-
> +// vim: set syntax=asciidoc:
> +
> +[[filesystem-custom]]
> +Customizing the generated filesystem images
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> ++Buildroot+ knows by default how to generate a few different kind of
> +filesystems, such as +squashfs+, +ext2/3/4+, +cramfs+... But those
> +filesystems are all generated to contain the complete target directory
> +hierarchy in a single filesystem, mounted as the root filesystem +/+.
> +That is, even if you select both an +ext2+ and a +squashfs+ filesystems,
> +the content of the two generated images will be the exact same, only the
> +types of the filesystems will be different.
> +
> +Most devices require a more complex setup, with different parts of the
> +directory structure split across different filesystems, each stored on
> +different partitions of one or more storage devices.
> +
> ++Buildroot+ can generate such complex setups, using a +partition table layout
> +description+. This is a simple text file, not unlike the +.ini+ style of
> +configuration files, that describes how the target directory hierarchy has
> +to be split across the target storage devices. It is a bit like a flattened
> +tree of the storage layout.
> +
> +Set the variable +BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE+ to the path of
> +the file containing your +partition table layout description+.
> +
> +See xref:part-layout-desc-syntax[] for the complete documentation of the
> ++partition table layout description+ syntax.
> +
> +[underline]*Note:* Although more versatile than the single filesystem image
> +mechanism, the +partition table layout description+ might be unable to
> +describe very complex setups. For example, it is not capable of handling
> +NFS-mounted filesystems, and +initramfs+ setups is not trivial (although
> +possible in most cases).
> diff --git a/docs/manual/customize.txt b/docs/manual/customize.txt
> index 7e46fd8..6d062ea 100644
> --- a/docs/manual/customize.txt
> +++ b/docs/manual/customize.txt
> @@ -14,6 +14,8 @@ include::customize-kernel-config.txt[]
>
> include::customize-toolchain.txt[]
>
> +include::customize-filesystems.txt[]
> +
> include::customize-store.txt[]
>
> include::customize-packages.txt[]
> diff --git a/docs/manual/partition-layout.txt b/docs/manual/partition-layout.txt
> new file mode 100644
> index 0000000..092ae1b
> --- /dev/null
> +++ b/docs/manual/partition-layout.txt
> @@ -0,0 +1,298 @@
> +// -*- mode:doc; -*-
> +// vim: set syntax=asciidoc:
> +
> +[[part-layout-desc-syntax]]
> +
> +Partition table layout description syntax
> +-----------------------------------------
> +
> +The +partition table layout description+ syntax is not unlike the standard
> +https://en.wikipedia.org/wiki/.ini[+.ini+] syntax. There are two types of
> +entries: +sections+, that may each contain zero or more +properties+.
> +
> ++Sections+ are specified between square brackets +[]+, _eg._: +[name]+.
> +
> ++Properties+ are specified as key-value pairs, _eg._: +key=value+, and
> +are documented as:
> +
> +* +key-name+ (optional or mandatory): description
> +** +value1+: description
> +** +value2+: description
> +** ...
> +
> +[underline]*Note:* Unlike the standard +.ini+ syntax, the +partition table
> +layout description+ _is_ case-sensitive.
> +
> +The order of +sections+ is irrelevant. However, for readability, we recomend
> +the +partition table layout description+ starts with the +global+ section.
> +
> +The global section
> +~~~~~~~~~~~~~~~~~~
> +
> +The +[global]+ section defines some global settings, and the list of devices.
> +
> +Properties for the global section
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +extract+ (mandatory): the type of base image to extract
> +** +tar+: extract the +rootfs.tar+ base image generated by +Buildroot+
> +
> +* +devices+ (mandatory): the comma-separated list of storage devices to
> + use on the device. Each device is the filename of the device node
> + present in +/dev+
> +
> +* +keep_partitions+ (optional): also copy the individual partition images
> + of all devices to +$(BINARIES_DIR)+. Settings from the device sections
> + or the partition sections take precedence over this one.
> +** +yes+: copy the individual partition images
> +** +no+ (the default): do not copy individual partition images
> +
> +The devices and partitions sections
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The devices and partitions sections define, for each device or partition,
> +the content of that device or partition.
> +
> +For each device listed in the +[global]+ section, there must be a
> +corresponding section named after that device.
> +
> +For each partition listed in a device section, there must be a corresponding
> +section named after that partition.
> +
> +Properties for the device section
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +type+ (mandatory): the type of content for that device or partition
> +** +boot+: the device contains one or more partitions, and
> + _may_ serve as a boot device
> +
> +* +keep_partitions+ (optional): also copy the individual partition images
> + for this device to +$(BINARIES_DIR)+. Settings from the partition
> + sections take precedence over this one.
> +** +yes+: copy the individual partition images
> +** +no+ (the default): do not copy individual partition images
> +
> +Properties for the partition section
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +type+ (mandatory): the type of content for that device or partition
> +** +fs+: the partition contains a filesystem
> +
> +* +size+: the size of that partition, in bytes
> +
> +* +keep+ (optional): copy this partition image to +$(BINARIES_DIR)+
> +** +yes+: copy this partition image
> +** +no+ (the default): do not copy this partition image
> +
> +Properties for +type=boot+
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +boot_type+ (mandatory): the partitioning scheme to use on this device
> +** +mbr+: use an https://en.wikipedia.org/wiki/Master_boot_record[MBR]
> + partitioning scheme
> +** +gpt+: use a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT]
> + partitioning scheme
> +
> +* +partitions+ (mandatory): the comma-separated list of partition(s) on
> + this device; no two partitions may have the same name, even if they
> + reside on different devices; partitions names shall match this regexp:
> + `^[[:alpha:]][[:alnum:]-_]*$` (_ie._ starts with a letter, followed by
> + zero or more alpha-numeric character or a dash or an underscore)
> +
> +* +partalign+ (optional): the alignment of partitions, in bytes; defaults
> + to an alignment of one, which means no alignment; depending on the
> + +boot_type+, some restrictions may apply, and are documented for each
> + +boot_type+
> +
> +Properties for +boot_type=mbr+
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +mbr_bootcode+ (optional): the bootcode to use, as a path to the file
> + containing the bootcode image, relative to the +$(BINARIES_DIR)+
> + directory; defaults to no bootcode (eg. filled with zeroes)
> +
> +* +partalign+: must be a multiple of 512
> +
> +Properties for +boot_type=gpt+
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +partalign+: must be a multiple of 512
> +
> +********
> +Currently, only 512-byte sectors are supported. 4k sectors are not.
> +********
> +
> +Properties for partitions whose containing device is +boot_type=mbr+
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +mbr_type+ (mandatory): the partition
> + https://en.wikipedia.org/wiki/Partition_type#List_of_partition_IDs[type]
> +
> +Properties for +type=fs+
> +^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +fs_type+ (mandatory): the type of filesystem to generate
> +** +ext+: generate an extended filesystem (ext2, ext3, ext4)
> +** +vfat+: generate a VFAT filesystem (FAT16, FAT32)
> +
> +* +fs_label+ (optional): the label to assign to this filesystem, if that
> + filesystem supports a label
> +
> +* +fs_files_0+, +fs_files_1+, +fs_files_N+ (optional): the list of files,
> + relative to $(BINARIES_DIR), to store in the filesystem. These entries
> + must be indexed starting from 0, and must be sequential: the first
> + missing entry ends the list
> +
> +* +fs_root+ (optional): the mountpoint of the filesystem
> +
> +* +fs_vfstype+ (optional): the type of filesystem to use when calling
> + +mount+, if different from +fs_type+
> +
> +* +fs_mntopts+ (optional): the mount options; defaults to +defaults+
> +
> +_Note_: valid use-cases for +fs_root+ and +fs_files_N+:
> +
> +* if only +fs_root+ is specified (and no +fs_files_N+): the filesystem
> + content is made exclusively from +$(TARGET_DIR)/$(fs_root)+, and the
> + filesystem is mounted at runtime
> +
> +* if both +fs_root+ and at least one +fs_files_N+ are specified: the
> + filesystem content is made exclusively from the +fs_files_N+ entries,
> + and mounted at runtime. +$(TARGET_DIR)/$(fs_root)+ must be empty
> +
> +* if at least one +fs_files_N+ is specified, and +fs_root+ is not: the
> + filesystem content is made exclusively from the +fs_files_N+ entries,
> + and the filesystem is not mounted at runtime
> +
> +* if neither +fs_root+ nor +fs_files_N+ is specified: this is an error
> +
> +Properties for +fs_type=ext+
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +ext_gen+ (mandatory): the generation of extended filesystem to generate
> +** +2+, +3+, +4+: for an ext2, ext3 or ext4 filesystem
> +
> +* +ext_rev+ (mandatory): the revision of the extended filesystem
> +** +0+ (ext2 only): generate a revision 0 extended filesystem filesystem
> +** +1+ (mandatory for ext3 or ext4): generate a revision 1 extended
> + filesystem
> +
> +Properties for +fs_type=vfat+
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +* +vfat_size+ (optional): the VFAT-size of the filesystem
> +** +12+, +16+, +32+: generate a FAT12, FAT16, or FAT32
> +
> +Generation of the filesystems
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The filesystems are generated in an order such that a filesystem that is
> +mounted as a sub-directory of another filesystem is generated first.
> +
> +A filesystem is filled with the content of the directory corresponding to
> +its mountpoint, and then that directory is emptied before continuing to the
> +next filesystem. That way, the mountpoints are empty before the filesystem
> +that contains them are generated.
> +
> +Finally, an entry in added in +/etc/fstab+ for each generated filesystem, so
> +they are mounted at boot time.
> +
> +Examples
> +~~~~~~~~
> +
> +.Simplest partition table layout description
> +====
> +----
> +[global]
> +extract=tar
> +devices=sda
> +
> +[sda]
> +type=boot
> +boot_type=mbr
> +partitions=root
> +partalign=1048576
> +
> +[root]
> +type=fs
> +fs_type=ext
> +fs_vfstype=ext4
> +fs_root=/
> +ext_gen=4
> +ext_rev=1
> +----
> +
> +The +partition table layout description+ above defines a single device
> ++sda+. That device contains a single partition, +root+, with an ext4
> +filesystem, which is filled with the whole content of the +rootfs.tar+,
> +and is mounted on +/+.
> +====
> +
> +.More complex table layout description
> +====
> +----
> +[global]
> +extract=tar
> +devices=mmcblk0,sda
> +
> +[mmcblk0]
> +type=boot
> +boot_type=mbr
> +partitions=boot,root
> +partalign=$((1024*1024))
> +
> +[sda]
> +type=boot
> +boot_type=mbr
> +partitions=data
> +partalign=4096
> +
> +[boot]
> +type=fs
> +mbr_type=$((0xC))
> +size=$((16*1048576))
> +fs_type=vfat
> +fs_mntopts=ro
> +fs_label=BOOT
> +fs_root=/boot
> +vfat_size=32
> +
> +[root]
> +type=fs
> +mbr_type=$((0x83))
> +size=268435456
> +fs_type=ext
> +fs_vfstype=ext4
> +fs_mntopts=discard,delalloc
> +fs_root=/
> +fs_label=ROOT
> +ext_gen=4
> +ext_rev=1
> +
> +[data]
> +type=fs
> +mbr_type=$((0x83))
> +size=$((4*1024*1048576))
> +fs_type=ext
> +fs_vfstype=ext2
> +fs_root=/data
> +fs_label=DATA
> +ext_gen=2
> +ext_rev=1
> +----
> +====
> +
> +The example above defines two devices, +mmcblk0+ and +sda+.
> +
> +The +mmcblk0+ device contains two partitions, +boot+ and +root+; partitions
> +are aligned on a 1MiB boundary. The +sda+ device contains a single partition,
> ++data+, aligned on a 4KiB boundary.
> +
> +The +boot+ partition is a 16MiB FAT32 filesystem filled with the content
> +of, and mounted on, +/boot+, and with label +BOOT+.
> +
> +The +data+ partition is a 4GiB ext2r1 filesystem filled with the content
> +of, and mounted on, +/data+, and with label +DATA+.
> +
> +The +root+ partition is a 256MiB ext4 filesystem filled the the rest of,
> +and mounted on, +/+, and with label +ROOT+.
How about comments ? Are they supported ?
In a next step, it would be really nice to have support for "human"
input for the different size, like 1G/M/K.
> diff --git a/fs/Config.in b/fs/Config.in
> index da4c5ff..44e04f7 100644
> --- a/fs/Config.in
> +++ b/fs/Config.in
> @@ -3,6 +3,7 @@ menu "Filesystem images"
> source "fs/cloop/Config.in"
> source "fs/cpio/Config.in"
> source "fs/cramfs/Config.in"
> +source "fs/custom/Config.in"
> source "fs/ext2/Config.in"
> source "fs/initramfs/Config.in"
> source "fs/iso9660/Config.in"
> diff --git a/fs/custom/Config.in b/fs/custom/Config.in
> new file mode 100644
> index 0000000..e5a8ee7
> --- /dev/null
> +++ b/fs/custom/Config.in
> @@ -0,0 +1,16 @@
> +config BR2_TARGET_ROOTFS_CUSTOM
> + bool "Custom partition table layout"
> + select BR2_TARGET_ROOTFS_TAR
> +
> +config BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE
> + string "path to the custom partition table layout description"
> + depends on BR2_TARGET_ROOTFS_CUSTOM
> + help
> + Enter the path to a partition-table for your device.
> +
> + This will allow Buildroot to generate a more complex target
> + image, which may consist of more than one filesystem on more
> + than one partition.
> +
> + See docs/manual/bla-bla on how to construct such a partition
> + table.
> diff --git a/fs/custom/boot/gpt b/fs/custom/boot/gpt
> new file mode 100644
> index 0000000..f978524
> --- /dev/null
> +++ b/fs/custom/boot/gpt
> @@ -0,0 +1,129 @@
> +# Build a complete GPT-based image
> +
> +#-----------------------------------------------------------------------------
> +DEPENDS+=( parted )
> +
> +#-----------------------------------------------------------------------------
> +# For a GPT-based partitionning, we need to compute the complete
> +# image size before we can attempt to generate the partition table.
> +# Then, we need to add the size of the GPT itself, plus that of its
> +# backup copy, plus the protective MBR.
> +# The size of the GPT itself depends on the sector size, and the
> +# number of partitions in the GPT. Sectors can be either 512-byte
> +# or 4096-byte large; The numbers of partitions is unlimited, but
> +# it is suggested there is space to store at least 128 of them; a
> +# partition description is 128-byte large.
> +#
> +# https://en.wikipedia.org/wiki/GUID_Partition_Table
> +#
> +# So, here's what we do:
> +# - consider 512-byte sectors (since GPT on 4k sectors is not well
> +# documented)
> +# - consider at least 128 partitions; if the layout defines more than
> +# that, we need to round that number up to the smallest multiple of
> +# 4 (since there are 4 partition descriptions in a 512-byte sector)
> +# - generate an empty, sparse file that is big enough to store the MBR,
> +# the two GPT copies, and the aligned partitions.
> +# - dump each partition in turn in their final location in that file
> +# - generate a parted script that creates the partition table in that
> +# file
> +#
> +# Simg = 512 + 2*(Sgpt) + Σ( aligned(Spart,512) )
Don't we want only ASCII char there ?
> +# Sgpt = 512 + Nent*128
> +#
> +# Where:
> +# Simg : size of the image
> +# Sgpt : size of one GPT
> +# Spart : size of each partition
> +# Nent : number of partition entries
> +# aligned() : the alignment function
> +#
> +# Sicne 4k-large sectors are not really explained on Wikipedia, we can
Small typo here.
> +# add this later on.
> +do_image() {
> + # ${1} is fs_root, irrelevant here
> + local img="${2}"
> +
> + # How many partitions do we have?
> + nb_parts=0
> + for part in ${partitions[${dev}]//,/ }; do
> + nb_parts=$((nb_parts+1))
> + done
> +
> + # How many partition entries do we need?
> + nb_entries=$((4*((nb_parts+3)/4)))
> + nb_entries=$((nb_entries<128?128:nb_entries))
> +
> + # The size of a single GPT
> + gpt_size=$((512+(128*nb_entries)))
> +
> + # Offset of the first partition
> + begin=$(align_val $((512+$(align_val ${gpt_size} 512))) ${partalign} )
> +
> + # Initialise our image file
> + dd if=/dev/zero of="${img}" \
> + bs=1 seek=${begin} count=0 \
> + conv=sparse 2>/dev/null
Why the redirect to /dev/null ? Don't we want to see if there was an
error ? I know it's unlikely to happen but you never know.
I have noticed you did several times so maybe there is a reason I am
not seeing here.
> +
> + # Compute the space required to store all partitions
> + # and store them in the image file
> + size_parts=0
> + _offset=${begin}
> + i=1
> + debug "adding partions descriptions\n"
> + for part in ${partitions[${dev}]//,/ }; do
> + debug " %s\n" "${part}"
> + _part_img="${tmp_dir}/${dev}.${part}.img"
> + _size=$( align_val $( stat -c '%s' "${_part_img}" ) 512 )
> + part_offset+=( ${_offset} )
> + _attr="${values["${part}:gpt_attr"]}"
> + _label="${values["${part}:gpt_label"]}"
> +
> + # If the partition has no label, use the filesystem label
> + if [ -z "${_label}" ]; then
> + _label="${values["${part}:fs_label"]}"
> + fi
> + if [ -z "${_label}" ]; then
> + _label="data"
> + fi
> +
> + debug " start=%s\n" "${_offset}"
> + debug " size =%s\n" "${_size}"
> + debug " end =%s\n" "$((_offset+_size-1))"
> +
> + dd if="${_part_img}" of="${img}" \
> + bs=512 seek=$((_offset/512)) \
> + conv=notrunc,sparse 2>/dev/null
> +
> + parted_script+=( mkpart "${_label}" \
> + ${_offset} \
> + $((_offset+_size-1)) \
> + )
> + if [ -n "${_attr}" ]; then
> + for attr in "${_attr//,/ }"; do
> + parted_script+=( set ${i} ${attr} on )
> + done
> + fi
> +
> + size_parts=$((size_parts+_size))
> + _offset=$((_offset+_size))
> + i=$((i+1))
> + done
> +
> + # Terminate our image file
> + img_size=$(align_val $(( begin + size_parts + gpt_size )) 512)
> + debug "begin =%s\n" ${begin}
> + debug "nb_entry=%s\n" ${nb_entries}
> + debug "gpt_size=%s\n" ${gpt_size}
> + debug "img_size=%s\n" ${img_size}
> + dd if=/dev/zero of="${img}" \
> + bs=1 seek=${img_size} \
> + count=0 conv=sparse 2>/dev/null
> +
> + for i in parted -s "${img}" mklabel gpt unit B "${parted_script[@]}"; do
> + debug "--> '%s'\n" "${i}"
> + done
> + parted -s "${img}" mklabel gpt unit B "${parted_script[@]}"
> +}
> +
> +# vim: ft=sh
> diff --git a/fs/custom/boot/mbr b/fs/custom/boot/mbr
> new file mode 100644
> index 0000000..667feed
> --- /dev/null
> +++ b/fs/custom/boot/mbr
> @@ -0,0 +1,64 @@
> +# Build a complete MBR-based image
> +
> +#-----------------------------------------------------------------------------
> +DEPENDS+=( genpart )
> +
> +#-----------------------------------------------------------------------------
> +do_image() {
> + # ${1} is fs_root, irrelevant here
> + local img="${2}"
> + local i begin part part_img size type _begin _size
> + local -a part_offset part_file bootcode
> +
> + # Fill-in the boot record
> + bootcode="${values["${dev}:mbr_bootcode"]}"
> + if [ -n "${bootcode}" ]; then
> + bootcode="${BINARIES_DIR}/${bootcode}"
> + else
> + bootcode="/dev/zero"
> + fi
> + debug "adding bootcode '%s'\n" "${bootcode}"
> + dd if="${bootcode}" of="${img}" bs=$((0x1be)) count=1 2>/dev/null
> +
> + # Generate partition entries
> + i=0
> + begin=${partalign}
> + debug "adding partitions descriptors\n"
> + for part in ${partitions[${dev}]//,/ }; do
> + debug " %s\n" "${part}"
> + part_offset+=( ${begin} )
> + part_img="${tmp_dir}/${dev}.${part}.img"
> + part_file+=( "${part_img}" )
> + size=$( align_val $( stat -c '%s' "${part_img}" ) 512 )
> + type="${values["${part}:mbr_type"]}"
> + # LBA is exressed in a number of 512-byte blocks
> + # and genparts only deals with LBA
> + _begin=$((begin/512)) # begin is already 512-byte aligned
> + _size=$((size/512)) # size is already 512-byte aligned
> + debug " start=%s (LBA %s)\n" "${begin}" "${_begin}"
> + debug " size =%s (LBA %s)\n" "${size}" "${_size}"
> + debug " type =%s\n" "${type}"
> + genpart -b ${_begin} -s ${_size} -t ${type} >>"${img}"
> + begin=$( align_val $((begin+size)) ${partalign} )
> + i=$((i+1))
> + done
> + nb_parts=${i}
> + # Generate entries for empty partitions
> + for(( ; i<4; i++ )); do
> + debug " (empty)\n"
> + genpart -t 0 >>"${img}"
> + done
> + # Dump the boot signature
> + printf "\x55\xaa" >>"${img}"
> +
> + for(( i=0; i<nb_parts; i++ )); do
> + part_img="${part_file[${i}]}"
> + offset=${part_offset[${i}]}
> + _offset=$(( offset/512 )) # offset is already 512-byte aligned
> + dd if="${part_img}" of="${img}" \
> + bs=512 seek=${_offset} \
> + conv=notrunc,sparse 2>/dev/null
> + done
> +}
> +
> +# vim: ft=sh
> diff --git a/fs/custom/boot/pre-post b/fs/custom/boot/pre-post
> new file mode 100644
> index 0000000..af4bcf5
> --- /dev/null
> +++ b/fs/custom/boot/pre-post
> @@ -0,0 +1,14 @@
> +#-----------------------------------------------------------------------------
> +# No dependencies
> +
> +#-----------------------------------------------------------------------------
> +do_image_pre() {
> + :
> +}
> +
> +#-----------------------------------------------------------------------------
> +do_image_post() {
> + :
> +}
> +
> +#vim: set ft=sh
> diff --git a/fs/custom/custom.mk b/fs/custom/custom.mk
> new file mode 100644
> index 0000000..ca53e0b
> --- /dev/null
> +++ b/fs/custom/custom.mk
> @@ -0,0 +1,25 @@
> +################################################################################
> +#
> +# custom partitioning
> +#
> +################################################################################
> +
> +# rootfs-custom uses rootfs.tar as the source to generate
> +# the resulting image(s), so we need to build it first.
> +ROOTFS_CUSTOM_DEPENDENCIES += rootfs-tar
> +
> +# If we are not selected, we won't have a partition table, so genimages
> +# will complain (both on stdout and return code), so it will faill, so
> +# we should not get our dependencies
> +ifeq ($(BR2_TARGET_ROOTFS_CUSTOM),y)
> +ROOTFS_CUSTOM_DEPENDENCIES += \
> + $(patsubst %,host-%,$(shell $(USER_HOOKS_EXTRA_ENV) fs/custom/genimages --show-depends \
> + '$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'))
> +endif
> +
> +define ROOTFS_CUSTOM_CMD
> + $(USER_HOOKS_EXTRA_ENV) fs/custom/genimages \
> + '$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'
> +endef
> +
> +$(eval $(call ROOTFS_TARGET,custom))
> diff --git a/fs/custom/fs/ext b/fs/custom/fs/ext
> new file mode 100644
> index 0000000..a184969
> --- /dev/null
> +++ b/fs/custom/fs/ext
> @@ -0,0 +1,26 @@
> +# Create an extended file system
> +
> +#-----------------------------------------------------------------------------
> +DEPENDS+=( e2fsprogs genext2fs )
> +
> +#-----------------------------------------------------------------------------
> +do_image() {
> + local root_dir="${1}"
> + local img="${2}"
> + local -a fs_opts
> + local gen rev
> +
> + fs_opts+=( -z )
> + fs_opts+=( -d "${root_dir}" )
> + [ -z "${size}" ] || fs_opts+=( -b $((size/1024)) )
> + [ -n "${ext_gen}" ] || ext_gen=2
> + [ -n "${ext_rev}" ] || ext_rev=1
> +
> + # Remember, we're running from Buildroot's TOP_DIR
> + GEN=${ext_gen} REV=${ext_rev} \
> + ./fs/ext2/genext2fs.sh "${fs_opts[@]}" "${img}" >/dev/null
> +
> + [ -z "${fs_label}" ] || tune2fs -L "${fs_label}" "${img}" >/dev/null
> +}
> +
> +# vim: ft=sh
> diff --git a/fs/custom/fs/pre-post b/fs/custom/fs/pre-post
> new file mode 100644
> index 0000000..d931437
> --- /dev/null
> +++ b/fs/custom/fs/pre-post
> @@ -0,0 +1,72 @@
> +#-----------------------------------------------------------------------------
> +# No additional dependencies
> +
> +#-----------------------------------------------------------------------------
> +do_image_pre() {
> + local i file
> +
> + # if fs_root_dir is not specified, we have to create one
> + # It *does* override the caller's fs_root_dir value, but
> + # that's on purpose
> + # If fs_root_dir is specified, and we have at least fs_files_0,
> + # then fs_root_dir/ must be enpty
> + if [ -z "${fs_root_dir}" ]; then
> + if [ -n "${values["${part}:fs_files_0"]}" ]; then
> + error "%s: no fs_root specified, and no fs_files_0\n" "${part}"
> + fi
> + fs_root_dir="$( mktemp -d "${tmp_dir}/XXXXXX" )"
> + else
> + if [ -n "${values["${part}:fs_files_0"]}" \
> + -a $( ls -1A "${fs_root_dir}" 2>/dev/null \
> + |wc -l \
> + ) -ne 0 ]; then
> + error "%s: %s is not empty, but fs_files_0 is specified\n" \
> + "${part}" "${fs_root_dir#${fs_root}}"
> + fi
> + fi
> +
> + i=0
> + while true; do
> + file="${values["${part}:fs_files_${i}"]}"
> + [ -n "${file}" ] || break
> + debug "%s: adding fs_files_%d %s\n" "${part}" ${i} "${file}"
> + install -D "${BINARIES_DIR}/${file}" "${fs_root_dir}/${file##*/}"
> + i=$((i+1))
> + done
> +}
> +
> +#-----------------------------------------------------------------------------
> +do_image_post() {
> + local rootfs_dir="${1}"
> + local fs_root="${2}"
> + local img_file="${3}"
> + local part="${4}"
> + local dev mntops vfstype fs_root_esc
> +
> + subname+="[post-image]"
> +
> + # Empty the partition's mountpoint
> + find "${fs_root_dir}" -maxdepth 1 \! -path "${fs_root_dir}" -exec rm -rf {} +
> +
> + # Add entry in fstab, but not if this is '/'
> + # Don't add either if rootfs was not extracted
> + if [ "${fs_root}" = "/" -o -z "${fs_root}" \
> + -o -z "${values["global:extract"]}" ]; then
> + return 0
> + fi
> + fs_root_esc="$( sed -r -e 's:/:\\/:g;' <<<"${fs_root}" )"
> + sed -r -i -e "/[^[:space:]]+[[:space:]]+${fs_root_esc}[[:space:]]/d" \
> + "${rootfs_dir}/etc/fstab"
> + dev="$( get_part_dev_node "${part}" )"
> + vfstype="${fs_vfstype:-${fs_type}}"
> + mntops="${fs_mntops:-defaults}"
> + printf "/dev/%s %s %s %s 0 0\n" \
> + "${dev}" "${fs_root}" \
> + "${vfstype}" "${mntops}" \
> + >>"${rootfs_dir}/etc/fstab"
> +
> + subname="${subname%\[post-image\]}"
> +}
> +
> +#-----------------------------------------------------------------------------
> +# vim: ft=sh
> diff --git a/fs/custom/fs/vfat b/fs/custom/fs/vfat
> new file mode 100644
> index 0000000..5fc243d
> --- /dev/null
> +++ b/fs/custom/fs/vfat
> @@ -0,0 +1,21 @@
> +# Create a VFAT file system
> +
> +#-----------------------------------------------------------------------------
> +DEPENDS+=( dosfstools mtools )
> +
> +#-----------------------------------------------------------------------------
> +do_image() {
> + local root_dir="${1}"
> + local img="${2}"
> + local -a fs_opts
> +
> + dd if=/dev/zero of="${img}" bs=${size} count=0 seek=1 2>/dev/null
> +
> + [ -z "${vfat_size}" ] || fs_opts+=( -F ${vfat_size} )
> + [ -z "${fs_label}" ] || fs_opts+=( -n "${fs_label}" )
> + mkfs.vfat "${fs_opts[@]}" "${img}" >/dev/null
> +
> + mcopy -i "${img}" "${root_dir}/"* '::'
> +}
> +
> +# vim: ft=sh
> diff --git a/fs/custom/genimages b/fs/custom/genimages
> new file mode 100755
> index 0000000..204f13b
> --- /dev/null
> +++ b/fs/custom/genimages
> @@ -0,0 +1,342 @@
> +#!/bin/bash
> +
> +#-----------------------------------------------------------------------------
> +main() {
> + local part_table="${1}"
> + local tmp_dir
> + local rootfs_dir
> + local -a devices
> + local extract
> + local cur_section
> + local -a sections devices partitions
> + local -A variables values partdevs
> + local sec dev part var val
> + local secs devs parts vars vals
> + local has_global_section
> +
> + # We need bash 4 or above for associative arrays
> + if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
> + error "bash 4 or above is needed\n"
> + fi
> +
> + if [ "${part_table}" = "--show-depends" ]; then
> + SHOW_DEPENDS=1
> + part_table="${2}"
> + trace() { :; } # Be silent, we just want the dependencies...
> + else
> + SHOW_DEPENDS=0
> + fi
> +
> + if [ ! -f "${part_table}" ]; then
> + error "'%s': no such file\n" "${part_table}"
> + exit 1
> + fi
> +
> + export PATH="${HOST_DIR}/usr/bin:${HOST_DIR}/usr/sbin:${PATH}"
> +
> + # Parse all the sections in one go, we'll sort
> + # all the mess afterwards...
> + debug "parsing partitions descriptions file '%s'\n" \
> + "${part_table}"
> + parse_ini "${part_table}"
> +
> + # The 'global' section is mandatory
> + has_global_section=0
> + for s in "${sections[@]}"; do
> + if [ "${s}" = "global" ]; then
> + has_global_section=1
> + break
> + fi
> + done
> + if [ ${has_global_section} -eq 0 ]; then
> + error "no global section defined\n"
> + fi
> +
> + # Create lists of devices, partitions, and partition:device pairs.
> + debug "creating intermediate lists\n"
> + devices=( ${values["global:devices"]//,/ } )
> + for dev in "${devices[@]}"; do
> + # Sanity check first: all devices must have a corresponding
> + # section, which means they should have a type
> + if [ -z "${values["${dev}:type"]}" ]; then
> + error "device '%s' has no type (no section?)\n" "${dev}"
> + fi
> + partitions+=( ${values["${dev}:partitions"]//,/ } )
> + for part in ${values["${dev}:partitions"]//,/ }; do
> + # Sanity check first: all partitions must have a corresponding
> + # section, which means they should have a type
> + if [ -z "${values["${part}:type"]}" ]; then
> + error "partition '%s:%s' has no type (no section?)\n" \
> + "${dev}" "${part}"
> + fi
> + partdevs+=( ["${part}"]="${dev}" )
> + done
> + done
> +
> + # Now, we must order the partitions so that their mountpoint
> + # is empty by the time we build the upper-level partition.
> + # For example, given this layout of mountpoints:
> + # /
> + # /usr
> + # /usr/var
> + # We must ensure /usr/var is empty at the time we create the /usr
> + # filesystem image; and similarly, we must ensure /usr is empty by
> + # the time we create the / filesystem image
> + # So, a simple reverse alphabetical sort will do the trick
> + debug "sorting partitions\n"
> + sorted_parts=( $(
> + for part in "${partitions[@]}"; do
> + # Partitions that are not mounted can be generated
> + # in any order
> + if [ -n "${values["${part}:fs_root"]}" ]; then
> + printf "%s:%s\n" "${part}" "${values["${part}:fs_root"]}"
> + else
> + printf "%s\n" "${part}"
> + fi
> + done \
> + |sort -t: -k2 -r \
> + |sed -r -e 's/:[^:]+$//;'
> + ) )
> +
> + # We do not want to create anything if we only want the dependencies
> + if [ ${SHOW_DEPENDS} -eq 0 ]; then
> + tmp_dir="${BUILD_DIR}/genimages.tmp"
> + rootfs_dir="${tmp_dir}/rootfs"
> + # Since we don't remove it in case of error (to be able to inspect its
> + # content), we must remove it now (a previous run may have left it).
> + rm -rf "${tmp_dir}"
> + mkdir -p "${rootfs_dir}"
> +
> + case "${values["global:extract"]}" in
> + tar)
> + # We must be root for the extract to work correctly
> + # (since it may have /dev nodes, or some files may
> + # belong to different users...)
> + if [ $(id -u) -ne 0 ]; then
> + error "error: not root\n"
> + fi
> + trace "extracting rootfs.tar\n"
> + tar xf "${BINARIES_DIR}/rootfs.tar" -C "${rootfs_dir}"
> + ;;
> + *) error "unknown extract method '%s'\n" "${extract:-(none)}"
> + ;;
> + esac
> + fi # ! SHOW_DEPENDS
> +
> + # Render all partition images
> + for part in "${sorted_parts[@]}"; do
> + trace "preparing filesystem for partition '%s'\n" "${part}"
> + render_img "${rootfs_dir}" "${part}" \
> + "${tmp_dir}/${partdevs["${part}"]}.${part}.img"
> + done
> +
> + # Aggregate all devices images
> + for dev in "${devices[@]}"; do
> + trace "assembling partitions in device '%s'\n" "${dev}"
> + render_img "${rootfs_dir}" "${dev}" "${tmp_dir}/${dev}.img"
> + done
> +
> + # If we need the dependencies, we can stop right now
> + if [ ${SHOW_DEPENDS} -eq 1 ]; then
> + return 0
> + fi
> +
> + # Copy all partitions and devices images to the image dir
> + if [ "${values["global:keep_partitions"]}" = "yes" ]; then
> + for part in "${sorted_parts[@]}"; do
> + debug "copying partition '%s' to image dir\n" "${part}"
> + dd if="${tmp_dir}/${partdevs["${part}"]}.${part}.img" \
> + of="${BINARIES_DIR}/$( get_part_dev_node "${part}" ).img" \
> + bs=4096 conv=sparse 2>/dev/null
> + done
> + fi
> + for dev in "${devices[@]}"; do
> + debug "copying device '%s' to image dir\n" "${dev}"
> + dd if="${tmp_dir}/${dev}.img" \
> + of="${BINARIES_DIR}/${dev}.img" \
> + bs=4096 conv=sparse 2>/dev/null
> + done
> +
> + [ -n "${DEBUG}" ] || rm -rf "${tmp_dir}"
> +}
> +
> +#-----------------------------------------------------------------------------
> +render_img() {
> + local rootfs_dir="${1}"
> + local img="${2}"
> + local img_file="${3}"
> + local type sub_type fs_root_dir
> +
> + type="${values["${img}:type"]}"
> + sub_type="${values["${img}:${type}_type"]}"
> +
> + # Sanity checks
> + [ -n "${type}" ] || error "'%s': unspecified type\n" "${img}"
> + if [ ! -d "fs/custom/${type}" ]; then
> + error "'%s': unsupported type '%s'\n" "${img}" "${type}"
> + fi
> + [ -n "${sub_type}" ] || error "'%s': unspecified %s_type\n" "${img}" "${type}"
> + if [ ! -f "fs/custom/${type}/${sub_type}" ]; then
> + error "'%s': unknown %s_type '%s'\n" "${img}" "${type}" "${sub_type}"
> + fi
> +
> + # Need to call the renderer in a subshell so that its definitions
> + # do not pollute our environment
> + subname="${sub_type}"
> + (
> + trap 'exit $?' ERR
> +
> + declare -a DEPENDS
> +
> + for var in ${variables["${img}"]//,/ }; do
> + eval "${var}=\"${values["${img}:${var}"]}\""
> + done
> + fs_root_dir="${rootfs_dir}${fs_root}"
> + . "fs/custom/${type}/pre-post"
> + . "fs/custom/${type}/${sub_type}"
> + if [ ${SHOW_DEPENDS} -eq 1 ]; then
> + for dep in "${DEPENDS[@]}"; do
> + printf "%s\n" "${dep}"
> + done
> + else
> + do_image_pre "${rootfs_dir}" "${fs_root}" "${img_file}" "${img}"
> + do_image "${fs_root_dir}" "${img_file}"
> + do_image_post "${rootfs_dir}" "${fs_root}" "${img_file}" "${img}"
> + fi
> + )
> + ret=${?}
> + [ ${ret} -eq 0 ] || exit ${ret}
Why not doing: [ ${ret} -ne 0 ] && exit ${ret} ?
> + subname=""
> +}
> +
> +#------------------------------------------------------------------------------
> +# Parse a .ini file
> +# $1: .ini file to parse
> +# The caller should define the following variables:
> +# sections : array
> +# variables : associative array
> +# values : associative array
> +# parse_ini() will fill those variables with:
> +# sections : the list of sections, one section per array index
> +# variables : the comma-separated list of varibles for a section,
> +# indexed by the name of the section
> +# values : the value of a variable in a section, indexed by the
> +# 'section:variable' tuple
> +# Eg.:
> +# sections=( [0]='section-0' [1]='section-1' )
> +# variables=( ['section-0']='var-0,var-1' ['section-1']='var-10,var-11' [...] )
> +# values=( ['section-0:var-0']='value-0-0' ['section-1:var-10']='value-1-10' [...] )
> +parse_ini() {
> + local ini_file="${1}"
> + local line var val
> + local cur_section
> + local var_sep
> +
> + while read line; do
> + line="$( sed -r -e 's/[[:space:]]*#.*$//; //d;' <<<"${line}" )"
> +
> + # Detect start of global section, skip anything else
> + case "${line}" in
> + "") continue;;
> + '['*']')
> + cur_section="$( sed -r -e 's/[][]//g;' <<<"${line}" )"
> + debug " entering section '%s'\n" "${cur_section}"
> + sections+=( "${cur_section}" )
> + continue
> + ;;
> + ?*=*) ;;
> + *) error "malformed entry '%s'\n" "${line}";;
> + esac
> +
> + var="${line%%=*}"
> + eval val="${line#*=}"
> + debug " adding '%s'='%s'\n" "${var}" "${val}"
> + var_sep="${variables["${cur_section}"]+,}"
> + variables+=( ["${cur_section}"]="${var_sep}${var}" )
> + values+=( ["${cur_section}:${var}"]="${val}" )
> + done <"${ini_file}"
> +}
> +
> +#-----------------------------------------------------------------------------
> +get_part_dev_node() {
> + local part="${1}"
> + local dev
> + local i c p
> +
> + dev="${partdevs["${part}"]}"
> + i="${values["${dev}:partstart"]:-1}"
> +
> + # If device node ends with a number, partitions are denoted
> + # with a 'p' before the partition number, eg.:
> + # /dev/mmcblk0 --> /dev/mmcblk0p1
> + # /dev/sda --> /dev/sda1
> + case "${dev#${dev%?}}" in
> + [0-9]) c="p";;
> + *) c="";;
> + esac
> +
> + for p in ${values["${dev}:partitions"]//,/ }; do
> + if [ "${p}" = "${part}" ]; then
> + printf "%s%s%d" "${dev}" "${c}" ${i}
> + return 0
> + fi
> + i=$((i+1))
> + done
> +
> + error "'%s': partition not found. WTF?\n" "${part}"
> +}
> +
> +#------------------------------------------------------------------------------
> +align_val() {
> + local val="${1}"
> + local align="${2}"
> + local aligned
> +
> + aligned=$(( ( (val+align-1) / align ) * align ))
> +
> + printf "%d" ${aligned}
> +}
> +
> +#------------------------------------------------------------------------------
> +# Some trace functions
> +_trace() {
> + local fmt="${1}"
> + shift
> +
> + printf "%s" "${myname}"
> + if [ -n "${subname}" ]; then
> + printf "(%s)" "${subname}"
> + fi
> + printf ": ${fmt}" "${@}"
> +}
> +
> +trace() {
> + _trace "${@}"
> +}
> +
> +debug() { :; }
> +if [ -n "${DEBUG}" ]; then
> + debug() {
> + _trace "${@}" >&2
> + }
> +fi
> +
> +error() {
> + _trace "${@}" >&2
> + exit 1
> +}
> +
> +on_error() {
> + local ret=${?}
> +
> + error "unexpected error caught: %d\n" ${ret}
> +}
> +trap on_error ERR
> +set -E -e
> +
> +#-----------------------------------------------------------------------------
> +export myname="${0##*/}"
> +
> +main "${@}"
> +
> +# vim: ft=sh
> --
> 1.8.1.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
More information about the buildroot
mailing list