Changes between Version 9 and Version 10 of doc/TorBOX/Dev/ArchivedDiscussion/SHELLSCRIPTS


Ignore:
Timestamp:
Apr 7, 2013, 5:49:23 PM (7 years ago)
Author:
proper
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • doc/TorBOX/Dev/ArchivedDiscussion/SHELLSCRIPTS

    v9 v10  
    562562 * (anonymous) that's done for t-g. TODO: T-W and we need documentation for aos-phyiso (or whatever we wanna call it)
    563563 * (anonymous) t-w done, we already tell about apt-get updates repeatedly.
     564
     565== [SHELLSCRIPTS] Move to file separated source [INFRASTRUCTURE] [DONE] ==
     566 * (adrelanos) I wanted to propose that after Whonix 0.2.0 but I think it's perhaps better to do it earlier. With torcheck and the problem to escape ' within ' ' it demonstrates, were the limits with one big shell script per VM are. It's possible to correctly escape it, but that makes the source less readable. Other than that, there are currently only two people who read and edit the script, proper and anonymous. I won't go as far as saying others can't understand it, but the effort must be high to get into it.
     567 * (adrelanos) I propose moving the files, i.e. the ones we paste using for example echo '...' > /etc/tor/torrc to separate pages/files. Not sure if that also hits the borders of "wiki development" or if something like git is required.
     568{{{
     569# Not complete yet, only for demonstration.
     570~/torbox_source_0.2.0/
     571~/torbox_source_0.2.0/tor-gateway.sh
     572~/torbox_source_0.2.0/tor-workstation.sh
     573~/torbox_source_0.2.0/tor-gateway/etc/torboxfirewall.sh
     574~/torbox_source_0.2.0/tor-gateway/etc/tor/torrc
     575~/torbox_source_0.2.0/tor-gateway/etc/resolv.conf
     576~/torbox_source_0.2.0/tor-gateway/usr/bin/leaktest
     577~/torbox_source_0.2.0/tor-workstation/usr/local/bin/uwt
     578~/torbox_source_0.2.0/tor-workstation/usr/local/bin/apt-get
     579~/torbox_source_0.2.0/tor-workstation/usr/local/bin/gpg
     580~/torbox_source_0.2.0/tor-workstation/usr/local/bin/...
     581# HARDCODED!
     582~/torbox_source_0.2.0/tor-workstation/home/user/.gnupg/gpg.conf
     583~/torbox_source_0.2.0/tor-workstation/home/user/.config/openbox/menu.xml
     584~/torbox_source_0.2.0/tor-workstation/usr/bin/arm.sh
     585~/torbox_source_0.2.0/tor-workstation/etc/network/interfaces
     586
     587# Perhaps there is also some shared code? Like delete_logs.
     588~/torbox_source_0.2.0/shared/delete_logs
     589}}}
     590   * (anonymous) I'd go with a "build_torbox" script in the root which creates tmp build directories and calls the other scripts, then sub-dirs for t-w and t-g and shared. We need separate scripts for building the basic debootstrap chroots and converting to VMI, configuring the chroot and finally to create the Vbox VMs.
     591    * (adrelanos) I guess VMI=VDI (virtualbox disk image)? Otherwise sounds good. The "build_torbox" script should support "build" and "clean" (delete temp dirs). Perhaps also do only one build step and/or repeat another one to ease debugging.
     592 * (adrelanos) Perhaps https://trac.torproject.org/projects/tor/wiki/doc/TorBOX/torbox_source_0.2.0/ or https://trac.torproject.org/projects/tor/wiki/doc/TorBOX/Dev/torbox_source_0.2.0/, i.e. https://trac.torproject.org/projects/tor/wiki/doc/TorBOX/torbox_source_0.2.0/tor-gateway/etc/torboxfirewall.sh.
     593 * (adrelanos) Not sure how to handle the >> files. To get this started we could still leave them in the tor-*.sh scripts.
     594 * (anonymous) Absolutely. But we need something better than trac to handle them, i.e. git. When (trademarkdomain user disappeared) comes online we could use a self hosted one like Tails does, or we use github (like Liberte).
     595 * (trademarkdomain user disappeared) Yes, I'm happy to implement an installation of Git on the (trademarkdomain user disappeared) server for our dev needs.
     596 * (anonymous) Great!
     597 * (anonymous) As long as we have no source repository we should keep as few scripts as possible for obvious usability reasons. Right now we have 5 scripts (t-g, t-w, t-debootstrap, t-chroot, torcheck). Any more absolutely required?
     598 * (adrelanos) Agreed and dunno for now.
     599 * (anonymous) 7 now, collected here: https://trac.torproject.org/projects/tor/wiki/doc/TorBOX/Dev#OurShellscriptsandBuildDocumentation
     600 * (anonymous) CLOSE? nothing to be done right now
     601 * (adrelanos) Done in github devel branch.
     602
     603== [BUG] [SHELLSCRIPTS] default color profile for gnome-terminal [FIXED ==
     604 * (anonymous) It currently defaults to black fonts on black background, not good.
     605 * (anonymous) I'm thinking of replacing it with xterm, the other terms just always suck one way or the other. gnome-term is nice, good features, but slow and got a horrendous config system. ~/.Xdefaults for xterm, run "xrdb .Xdefaults" to apply. xterm isn't "user friendly", but anyone who uses the terminal should be knowing what he's doing anyway. But at least copy and paste should be a bit more obvious for windows/mac users... If someone fixes the gnome-term .config that would be appreciated.
     606{{{
     607xterm*background:   white
     608xterm*foreground:   black
     609XTerm*utf8: 1
     610XTerm*eightBitInput: false
     611XTerm*eightBitControl: false
     612XTerm*eightBitOutput: true
     613}}}
     614 * (adrelanos) I am happy with any quick fix for Whonix 0.2.0. xterm lacks features, such as an tabbed interface but it's ok. The geeks are free to install a replacement.
     615 * (anonymous) so xterm config doesn't really work as I hoped either. It's going to be urxvt, if only temporary (apt-get install -no-install-recommends rxvt-unicode)
     616 * (anonymous) workaournd implemented but ticket is not solved, urxvt is no replacement for a user friendly terminal that at least supports copy and past with mouse context menu (and tabs would be nice too).
     617 * (anonymous) I think the roxterm segfault has been fixed in precise. If it's stable we got our replacement!
     618 * (anonymous) roxterm still crashes, so does terminator. Gnome-terminal is still the best option.
     619 * (anonymous) right now, it defaults to xterm, even with urxvt installed. UPDATE:or not
     620 * (adrelanos) Please try:
     621{{{
     622USERNAME="user"
     623sudo -u $USERNAME gconftool-2 --type string --set /apps/gnome-terminal/profiles/Default/foreground_color "#FFFFFFFFFFFF"
     624}}}
     625 * (adrelanos) Already added to the script. If it works for you, we can close this one.
     626 * (adrelanos) Unfortunately will not be fixed in [0.2]. When I found the command above it worked, but it didn't work within the script. Maybe gnome terminal has to be started at least once, closed and then the command will have an effect. Needs testing.
     627 * (adrelanos) Not one gconftool-2 command is required, it's three. Hopefully fixed in latest git devel branch.
     628 * (adrelanos) Now fixed, because no longer using Openbox.
     629
     630== [SHELLSCRIPTS] Ubuntu unattended (automatic) installation debootstrap vs preseed [DONE] ==
     631 * (adrelanos) That's the missing link before we can have fully automated builds. Looks feasible to implement for me.
     632 * (adrelanos) I found a [http://www.debuntu.org/how-to-unattended-ubuntu-network-install tutorial], which is pretty straight forward, explaining how to install Ubuntu unattended. There is an updated [https://help.ubuntu.com/12.04/installation-guide/example-preseed.txt Ubuntu 12.04 example-preseed.txt]. Preseed is the [https://help.ubuntu.com/12.04/installation-guide/i386/appendix-preseed.html recommend] way in the [https://help.ubuntu.com/12.04/installation-guide/i386/index.html Ubuntu 12.04 Installation Guide].
     633 * (adrelanos) Alternatively, there is also the [http://fai-project.org/ FAI] (Fully Automatic Installation) project, but the learning curve seams way too flat to me.
     634 * (adrelanos) Advantages of automatic installation:
     635  * (adrelanos) When it's done, it will be much easier, faster, secure and motivating to make new binary builds.
     636  * (adrelanos) More secure, because we eliminate the possibility to setup the wrong settings.
     637 * (adrelanos) Disadvantages of automatic installation:
     638  * (adrelanos) Who knows preseed syntax and unattended installation (possible less security reviews).
     639  * (adrelanos) Requires additional software for building Whonix, such as DHCP server, TFTP server. We have to configure them leak free: they may listen on localhost only. - If that's hard to configure, we simply use iptables as firewall and block all incoming ports. That may be unpleasant for people who want to build themselves and have server ports open. We recommend a dedicated build machine already.
     640  * (adrelanos) It requires the net installation (netboot.tar.gz).
     641  * (adrelanos) Optionally, hiding the fact, you are building Whonix could be difficult, when using netboot.tar.gz.
     642  * (adrelanos) More complex build script. It will be more difficult to manually build Whonix. After adding the command line commands for using VirtualBox by command line, it's already harder to figure out, which VirtualBox gui options to choose to do the same. Should we maintain two versions? Manual creation of VirtualBox VM's and manually choosing all options for Ubuntu installer?
     643 * (adrelanos) I'd like to hear some opinions, also open for alternatives...
     644 * (adrelanos) An alternative to preseed could be to manually create the image. Mount the image, chroot into it. I have no experience in that field. Dunno if that makes more sense. [http://wiki.debian.org/Debootstrap Debootstrap]
     645 * (anonymous) I found no helfpful docs on how to do that. I don't know how it could be possible. The problem is we needed to directly access the virtual disk in order to install grub to it. The only way I found is to use a nother VM, mount the drive there and run grub-install. That doesn't really help us... The most promising option I think is to remaster the install CD to include a preseed file and auto start syslinux:  http://g0blin.co.uk/programming/ubuntu-preseed-virtualbox-and-hetzner-eq4-virtual-machines/ We could already include the scripts too and copy them somehow to the target systems so we don't need ssh and port forwarding.
     646 * (adrelanos) Nice link. The most complicated thing with preseed.txt is getting the CD to read it. He remasters the CD instant of using a DHCP and tftp server. Tempting. I did already struggle with tftp package bugs and the DHCP/tftp setup is overly complex.
     647 * (adrelanos) grub-install should not be a problem. Linux is very flexible. With wubi you can even boot from NTFS (an image gets booted from a file, loop device). Once an image is mounted, there should be no relevant difference from a real hdd. UPDATE: [https://github.com/ra--/Tor-gateway/blob/master/build.sh Image generator example], looks easy.
     648   * (anonymous) I don't see how that helps with building a vdi from a chroot "folder", the OpenWrt-ImageBuilder archive already contains grub stages and squashfs and everything nicely prepared. The script itself may look easy at first glance but it's just a wrapper for the OpenWrt scripts.  If we can mount the vdi from the chroot, that would work. But that doesn't seem [http://superuser.com/questions/34913/mount-virtualbox-vdi-disk possible] (vdfuse is not in ubuntu); [http://muralipiyer.blogspot.com/2008/02/mounting-virtualbox-vdi-disk-authentic.html vditool is gone], [http://serverfault.com/questions/210684/how-do-you-mount-a-vdi-image-on-linux qemu-ndb doesn't work for me]. There most be a working solution... Another way would be to create raw or other image that can be mounted and then convert that to vdi. Update: got it working, nbd software was missing.
     649 * (adrelanos) We have several options. 1) Remaster the CD, 2) preseed 3) mount, Debootstrap/Multibootstrap or similar. We should compare simplicity, flexibility, leaks, distribution agnostic (easy switching to another distro), and which solution is the least hackish (also supported in a few years). If you could help with any of the methods, that were of course also a strong point.
     650    * (anonymous) remastering and preseed is distro specific but pretty much every big distro offers similar functionality. preseed with netboot just doesn't sound that secure, chrooting and building that way is the most flexible but definitely more complicated. It's also slow if done over Tor because it fetches everything via network each time you build it as opposed to use packages supplied by the install iso.
     651    * (anonymous) https://trac.torproject.org/projects/tor/wiki/doc/TorBOX/Dev/Debootstrap I have no further interest in messing around with GRUB, I spent several hours (testing it is a chore, even with VMs) on trying to get it working but to no avail. I think we need to look into transforming a loopback into a vdi but I don't even know if grub2 will handle that. I always just get "/dev/something does not have any corresponding BIOS drive." when running grub-install. Is there no frigging command line anymore? grub-legacy should work but I fear support is going to be dropped by most distros, If it wasn't for the updating issue I'd just go with syslinux and create isos,,,
     652     * (adrelanos) VBOX [https://www.virtualbox.org/manual/ch05.html#vdidetails supports] VDI, VMDK, VHD, and some HDD (forget about HDD) as disk file images. Perhaps this can be of any help.
     653 * (adrelanos) Maybe of any help: [http://wiki.debian.org/VMBuilder VMBuilder]; [https://trac.torproject.org/projects/tor/wiki/doc/SetupTorDAwithKVM Setup Tor DA with KVM]; [http://nicolas.barcet.com/drupal/ubuntu-dev-week-vmbuilder ubuntu-dev-week-vmbuilder]
     654 * (anonymous) Didn't look into it (some links seem dead, vmbuilder only mentions KVM) but from http://ebroder.net/2009/08/04/installing-grub-onto-a-disk-image/ : "For various reasons, I wasn’t able to use the multitude of VM installers already out there, but one thing I noticed is that most of them don’t actually install a bootloader..."
     655  * (adrelanos) All links up again. Yes KVM and only less important virtualizers only. After switching to Virtual Box command line, I would rejoice trashing that work. But I'd switch to a better supported virtualizer anyway. KVM is build in linux kernel and more supported. We just haven't researched if it's suited for Whonix (hide hardware serials etc.). And also the userinterface virt-manager is not so pretty. Supports only VNC and no "direct output". Looks like KVM is more suited for servers but not as client operating system, browser through VNC is not so much fun.
     656 * (anonymous) Debootstrap can be used to build Vbox images but we need to investigate host related leaks. Liberte for example can apparently be built behind tor by setting a proxy variable (didn't test it, the whole build takes ages). Does that take care of resolv.conf, dhcp and so on?
     657  * (adrelanos) Very good questions. How will we find out? Shall we ask the debootstrap devs?
     658 * (anonymous) I don't like the idea of building vbox in vbox because of performance reasons but also how would one bootstrap such system? The very first t-w one builds needs to be built outside of a t-w...
     659  * (adrelanos) I understand your concern, those are valid. Performance is indeed poor. It's a real waste of time. Development were faster if they run faster. Bootstrapping is also complicated. The existing devs already have a working Whonix, but new ones? Two solutions: 1. we find a packager who does not require to stay anonymous (low probability) or 2. provide instructions to build a minimal build system manually (as we had in past). Update: Building manually will be easy as soon we move to more file separated source.
     660 * (anonymous) with debootstrap at least one doesn't have to run nested VMs for the heavy lifting. VBox is only needed for configuring the VM settings and then exporting the already prepared vdi.
     661  * (adrelanos) Yes. Good. Controlling things inside VBox would have been a mess anyway.
     662 * (anonymous) basics are done and working. Missing stuff: debconf dialog for certain packages, virtualbox scripting and export. that can all be found in TODOs directly in the script. ticket can be closed?
     663 * (adrelanos) I can't yet fully understand the new scripts you created. Not your fault! I just have to to read more manuals etc, there are a lot changes in a very short time. Thanks for that work. This is no criticism.
     664 * (adrelanos) That said, can you update BuildDocumentation as well? Imho now it's not clear how to build. I think the manual installation followed by a T-* script should be deprecated. BuildDocumentation should walk through the automated build method.
     665 * (adrelanos) I am not sure debootstrap is the right way to go. I spend hours without significant progress. There are too much strange bugs. /var/run not mounted on shutdown/restart (ok, thats maybe minor). dpkg-reconfigure -a hangs on adduser (can always press yes/no, but change will not be recognized). This bug is so seldom, you don't even find it on google. Even if reported, I expect it to be ages until fixed. We're attempting what [https://launchpad.net/ubuntu/+source/vm-builder/ vmbuilder] did when jeos was still an active project. That project is now inactive and full of blocking bugs. (I couldn't get build virtualbox images, perhaps someone else can get it.) When the project was still current, they had way more developers (some with @ubuntu mail address). Debootstrap/chroot is (almost) equally hard as linux from scratch and this is far outside my skill set. ubuntu-vm-builder (python-vm-builder) also supports building kvm, vmware server, vmware 6 or xen images. Some of those formats are probable working better than virtualbox format. I tried building a vmware server image (vdmk) and simply used it for Virtual Box. I couldn't imagine that beside the disk format and the description file, the content of the image differs. That's wasn't a good hack either, also dpkg-reconfigure -a is broken and even apt-get update will show hdd kernel errors. Imho we can't do the "vmdebootstrap thing" our self. There are some alternative projects. https://gitorious.org/vmdebootstrap http://wiki-oar.imag.fr/index.php/Kameleon maybe others. I'd like to carefully choose the best one of them. I am willing to switch to KVM and/or Debian Squeeze (if in mid term update to Wheezy might work) right away if necessary.
     666 * (adrelanos) And even if we get vm-de-bootstrap or our own debootstrap or anything else running... It's still a custom hack. As soon the system makes any trouble you have some custom hacked up installation. Very very few people go this way = very badly supported, very hard to find solutions. Perhaps we should go back to preseed, that's virtually using the installer without user interaction. The system will be a "standard" one.
     667 * (adrelanos) I am trying [http://askubuntu.com/questions/122505/how-do-i-create-completely-unattended-install-for-ubuntu this] (with kickstart only, preseed might be broken for Ubuntu 12.04, still investigating). New plan:
     668{{{
     669build script will run the following scripts... Easy to enable/disable steps for debugging.
     670https://trac.torproject.org/projects/tor/wiki/doc/TorBOX/Dev/Build
     671download and verify cd https://trac.torproject.org/projects/tor/wiki/doc/TorBOX/Dev/GetISO
     672modify iso (preseed / kickstart) https://trac.torproject.org/projects/tor/wiki/doc/TorBOX/Dev/ModifyISO
     673create vm / start vm / install vm (without network card) https://trac.torproject.org/projects/tor/wiki/doc/TorBOX/Dev/CreateVM
     674mount virtual hdd / ChangeRoot / leave chroot / unmount virtual hdd
     675https://trac.torproject.org/projects/tor/wiki/doc/TorBOX/Dev/ChangeRoot
     676T-G script https://trac.torproject.org/projects/tor/wiki/doc/TorBOX/Dev/TGScript
     677T-W script https://trac.torproject.org/projects/tor/wiki/doc/TorBOX/Dev/TWScript
     678export vm
     679cleanup/clean
     680}}}
     681 * (adrelanos) This is probable are more realistic plan and much easier to implement.
     682 * (anonymous) about dpkg-reconfigure: if you report such bugs on launchpad, I doubt you'd get much feedback. Debian cares a bit more and since it's the upstream for 'deb'ootstrap I think there are good chances it will be fixed. The debian iso as well as Ubuntu server are actually using debootstrap too... It's probably just some missing package. Nothing really broken either, the configuration is still set. /var/run could be a mtab/fstab problem. What other bugs are there? The debootstrapped images work, there can only be minor problems and they can all be fixed. Really, installing via iso and debootstrap is the same thing techincally. Just one of the two is muuuch faster.
     683 * (adrelanos) Dunno, don't remember very well anymore. The dpkg-reconfigure bug, the "console-utilities console-tools console-data something broken" bug, and 'locale' returned an error, "Scroll wheel in VM" bug. Perhaps that was it. Unfortunately, I really was unable to fix them. That's why I worked on the preseed method, btw I am almost done and can build and release them. Creating images with debootstrap without running something in VMs is certainly more elegant. Using preseed has the advantage that that VM never gets access to /dev/..., therefore less likely to leak information into VM.
     684 * (adrelanos) Removed [0.2] tag because I am going to release 0.2 using preseed method.
     685 * (adrelanos) I am currently testing [http://wiki-oar.imag.fr/index.php/Kameleon Kameleon]. Although it's from 2010, first impression is very good. Looks like it needs to run on Debian to create a Debian VM, but switching to Debian might be a good idea so or so.
     686 * (adrelanos) Another interesting similar tool for this purpose: [http://aeolusproject.org/oz.html Oz is a tool for automatically installing guest OSs with only minimal up-front input from the user.]; [https://rwmj.wordpress.com/2011/08/30/playing-with-oz-install/ Playing with oz-install]; [https://github.com/clalancette/oz on github].
     687 * Done. Switched to grml-debootstrap/chroot. Perhaps in future to kameleon if the adduser bug with "dpgk-reconfigure -a" is gone.
     688