X-Git-Url: https://git.tld-linux.org/?a=blobdiff_plain;f=pld-builder.new%2Fdoc%2FARCHITECTURE;fp=pld-builder.new%2Fdoc%2FARCHITECTURE;h=0000000000000000000000000000000000000000;hb=3dc53b6865f716904ad4d02e934e04b3d4855c32;hp=7bcbcefd75ed610ed6c53c6db168b3ea03132ec9;hpb=b235722fd546915a60800f2660d76f684d6f1445;p=TLD.git diff --git a/pld-builder.new/doc/ARCHITECTURE b/pld-builder.new/doc/ARCHITECTURE deleted file mode 100644 index 7bcbcef..0000000 --- a/pld-builder.new/doc/ARCHITECTURE +++ /dev/null @@ -1,165 +0,0 @@ -0. ABSTRACT - -Below is a description of the way the builder infrastructure works. You should -read it in order to be able to understand what's exactly going on when you try -to build a package. - -(It assumes you have a basic understanding of how RPM packages are created.) - -1. OVERVIEW - -The builder infrastructure is divided into three parts: -- the client -- the source builder -- the binary builders - -The basic idea is to allow developers (requesters) to easily specify which -package they want built, and have the rest of the system take care of building -and upgrading packages, uploading them where appropriate and reporting back to -the developers on the results (were there errors, how long it took to build, -etc.). - -What follows is a more detailed description of what each part is supposed to -do. - -1.1 NOTES ON UPLOADING - -When talking of uploading packages and logs it is meant as placing them -somewhere on the original system (via a simple 'cp'; this also includes copying -to /dev/null), or placing them on a remote system via the scp or rsync -protocols. - -1.2 TOOLS USED - -99% of the system is written in python, with remaining parts using shell -scripts (POSIX shell is assumed, bashisms are to be shot at sight and their -authors beheaded in a public display). The presence of various helper tools is -assumed. They are: gpg, rsync and scp command line utilities. - -2. SHORT DIAGRAM - -Below are complete basics of how data is tossed around. Only the most relevant -transfers are presented. - - status info status info - +--------------+<--+------------------+ - | build | | | - V request | V src.rpm | -requester ---------> src.builder ---------> bin.builders - | | - | arch.rpms | - FTP server <---+------------------+ - -3. CLIENT - -Main task: sending build requests to source builders. - -This is the simplest part. It consists of a shell script (make-request.sh), -which, based on the way it's configured and on the cmdline parameters given to -it, generates an xml-formated build request, which it then signs with the -requester's PGP key (using the gpg utility) and then sends it to the source -builder via a sendmail compatible command line application (by default invoking -"sendmail -t"). - -Two modes of operation are: -- sending requests to build chosen package(s) on a specified group of builders -- sending a chosen command to be executed on a specified group of builders - -See HOWTO-CLIENT for a hands-on approach to setting yourself up for being able -to send out build requests. - -4. SOURCE BUILDER (AKA SRC.BUILDER) - -Main task: building src.rpm packages and serving them to binary builders. -Additional tasks: reporting back to the requester about the status of the build -and uploading generated src.rpm packages where appropriate. Also -- tagging, -access control, uploading buildlogs and keeping the 'queue.html' file in sync. - -The source builder is the central place for managing access rights for -requesters. Upon receiving a request (via email), it first checks the pgp -signature with which the request was signed, against the pgp key currently -assigned to the requester (all done with the 'gpg' command). After verifying -that the requester name/requester pgp signature pair match, it then checks if -the requester is actually allowed to perform the actions she wants to perform. -That is: does she have access to a specified binary builder, is she allowed to -specify the 'upgrade' flag or not, is she allowed to use the priority she set -for the request and even if she is allowed to request building of a given -package. - -Once a request passes all verifications, the src.builder uses its own chroot -equipped with a basic build environment to fetch the sources of a given -package, tag them if configured to do so and create a src.rpm out of them (all -done by invoking the 'builder' script as found in the SPECS module of the PLD -cvs repository (see http://www.pld-linux.org/Repositories for details)). - -After finishing the build procedure the src.builder reports (via email) to the -requester on the status of the build (OK/FAIL), updates the 'queue.html' file -with that information (urls for PLD2.0 and 3.0 queues: -http://ep09.pld-linux.org/~buildsrc/queue.html and -http://src.th.pld-linux.org/queue.html) and, if configured to do so, -uploads the complete log of the whole procedure (called the buildlog) -somewhere, most likely to our PLD buildlogs system -(http://buildlogs.pld-linux.org). - -If the build procedure was successful, the resulting src.rpm file is most -likely uploaded somewhere (if configured to do so; in our case it's our FTP -server) and, what is most important, it's being served via a http server for -the binary builders to download. An xml-formated (and pgp signed) 'queue.gz' -and a small 'max_req_no' files are also updated with relevant information, in -order for the bin.builders to be able to notice new requests and figure out -which are addressed to which builders. - -The last matter taken care of by the src.builder is receiving (gpg signed) -emails from bin.builders with status information regarding what happened to -various builds (whether they succeeded or failed) and updating the 'queue.html' -file accordingly. - -See the HOWTO-BUILDERS file for a hands-on approach on setting up builders. All -of the technical details are there. - -5. BINARY BUILDERS (AKA BIN.BUILDERS) - -Main task: building arch.rpm packages and uploading them where appropriate. -Additional tasks: reporting back to the requester and the source builder on the -status of the build. Also -- uploading buildlogs. - -Upon noticing that a new request has been published by the src.builder and -verifying the appropriate gpg signatures, binary builders download the relevant -src.rpm files and use their build-environment-equipped chroots to start -building them (by invoking the 'rpmbuild' command). Once a build is completed -(meaning arch.rpm files are produced, or the whole procedure is interrupted for -whatever reason), the status of the build (OK/FAIL) is sent via email both to -the requester and to the src.builder. - -Procedure for uploading the produced arch.rpms and buildlogs is the same as -with the src.builder. - -Bin.builders however need to deal with one thing src.builders are mostly not -affected by -- managing the build environments inside their chroots. That means -installing new packages if required by a given build (using the 'poldek' -package manager; http://poldek.pld-linux.org/), upgrading older packages for -much the same reasons, downgrading from time to time and locally caching newly -built files for short periods of times. - -All of this, along with lots of quirks and obscure details is explained in the -HOWTO-BUILDERS file. - -6. Q/A - -Q: What's this 'tagging' all about with the src.builder? -A: When a src.rpm is created and arch.rpms built based on it, it is sometimes -useful to be able to check which sources were used to build them. Casual users -might be satisfied with just extracting everything from the src.rpm (that is -what they are uploaded to the ftp server for), but the old ones aren't always -available and even if they are, developers require source control in order to -work efficiently. Hence all sources used to build a given src.rpm for a given -line are tagged appropriately in the PLD CVS repository, for example trying to -build cthulu.spec on the 2.0 (codename Ac) line of our distro might result in a -tag looking something like this being set -- 'auto-ac-cthulu-6.6-6'. -(Technically the previous sentence is not 100% correct due to (a) the way our -DISTFILES infrastructure impacts which sources are and aren't in the CVS repo -and hence are subject to tagging and (b) specific policies regarding tagging -for each distro line; for the latter refer to appropriate policies.) - - -# vi: formatoptions=aw expandtab