Couldn't avr-libc be an "improper" library?

Go To Last Post
44 posts / 0 new
Author
Message
#1
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I mean, couldn't it be distributed in source form, together with some user-friendly way to build it for a given AVR model at the moment it is needed for the first time. Eventually, a "descruction" ("clean") could be provided too, for models which are not needed anymore.

I admit I did not think much on the details, and I understand they are not trivial.

The benefit would be
- probably much smaller size
- more compressible (even smaller size for an "installer", i.e. less internet bandwidth - I know the westerner tend to neglect this as an argument these days)
- could be more tightly tailored to individual AVRs (now, some of the lesser important features are a tradeoff for a group, e.g. the memory sizes in makefile)

Just a wild idea, and I would like to hear the strong opinions... :-)

Jan Waclawek

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

For me it is not clear, what you want.
I thought the linker would remove all functions which are not referenced. Or is this not the case?

Isn't the code allready optimized in assembly to be rather small and fast?

In the beginning was the Word, and the Word was with God, and the Word was God.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I am talking about the size of the library, not the application's binary. Again, a few hundreds of megabytes to be downloaded and installed doesn't bother you perhaps, given your location, but I still find it a waste of resources.

There's an other side of the coin, too: in the last months' of avr-libc we've seen quite a couple of bugs (delay.h, rjmp in log, _PF() functions, to name a few) which are relatively easy to fix through minor edits and local recompilation.

JW

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:
a few hundreds of megabytes to be downloaded and installed

huh? It looks to me like the current avr-libc is only 1.4MB (download size. Expands to about 20MB.)

Presumably what you're talking about could be achieved by downloading the source instead of the binaries and building locally. I don't know if the source download mechanisms (svn) compress things as much as the binaries...

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

I don't know if the source download mechanisms (svn) compress things as much as the binaries...

Perhaps not, but OTOH after you've created your initial working copy then subsequent updates would only d/l "the delta" or at least only the changed files.

As of January 15, 2018, Site fix-up work has begun! Now do your part and report any bugs or deficiencies here

No guarantees, but if we don't report problems they won't get much of  a chance to be fixed! Details/discussions at link given just above.

 

"Some questions have no answers."[C Baird] "There comes a point where the spoon-feeding has to stop and the independent thinking has to start." [C Lawson] "There are always ways to disagree, without being disagreeable."[E Weddington] "Words represent concepts. Use the wrong words, communicate the wrong concept." [J Morin] "Persistence only goes so far if you set yourself up for failure." [Kartman]

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

westfw wrote:
Quote:
a few hundreds of megabytes to be downloaded and installed

huh? It looks to me like the current avr-libc is only 1.4MB (download size. Expands to about 20MB.
Where do you have an avr-libc binary installation from?

In WinAVR20100110, the /avr/include is 7M, /avr/lib is 10M, /doc/avr-libc is 5M, that for me sums up to 22M plus 1M the small files' penalty. They collectively zip to 5.5M (I know zip is not the best packer but it is not a bad benchmark). 7zip using bzip2 packing compresses it to some 3M.

Now forward one and half year, get the current Atmel's "toolchain". /avr/include is now 9M, /avr/lib is 18M (new devices added), /doc/avr-libc is 7M (after having thrown out the man subdirectory to compensate somewhat for incompetence of the Atmel crew; I refuse to investigate further the source of inflation), zips to 7M.

Now let's talk also about /lib/gcc/avr/ - yes I know it is not avr-libc, but it has the same "per architecure" structure and is plagued by the same rigidity as avr-libc. Add another few Ms.

So, okay, it's not hundreds of megs "only" ten or tens, but it's steadily growing for little reason, and there's still the flexibility thing.

I also would expect it to be primarily distributed as a package with the compiler as it is the practice now, so svn packing or not is not an issue. As Johan said above, the deltas after the initial installation are truly negligibly small - and only a fraction of users who really need them would download them anyway.

I repeat, I am not pushing this thing, just want to discuss it. Thanks.

Jan

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I don't believe the build system for avr-libc supports per-device compilation. IIRC Eric was interested in changing it though.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

atomicdog wrote:
I don't believe the build system for avr-libc supports per-device compilation.

That would need to be done indeed. And perhaps more, if it is to be truly user friendly.

atomicdog wrote:
IIRC Eric was interested in changing it though.
That would result in true explosion of the binaries' size.

JW

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:
Where do you have an avr-libc binary installation from?

Straight from the project at savannah:
http://download.savannah.gnu.org...
avr-libc-1.7.2rc2252.tar.bz2 22-Sep-2011 07:40 1.4M

17.8MB expanded; my doc directory is significantly smaller than you quote (~1MB) (I don't know why.)
(it doesn't surprise me a lot that tar.bz is smaller than zip-like archives. It probably saves a lot of dictionary restarts...)

I dunno. I've built gcc from source; it's a pain in the neck; dependency hell. And the size of your AVRStudio5 download isn't going to be dominated by the size of avr-libc, or even the complete WINAVR toolset.
I sympathize with the lower-bandwidth problems, but arv-libc looks like the wrong point to address.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I'm still not getting what the actual motivation here is? Surely download bandwidth is no longer considered the prime motivator (if it is someone tell Atmel about the > 0.5GB behemoth!). So is this just about being able to apply patches locally to fix/update the lib rather than waiting for the next distribution?

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

westfw wrote:
Quote:
Where do you have an avr-libc binary installation from?

Straight from the project at savannah:
http://download.savannah.gnu.org...
avr-libc-1.7.2rc2252.tar.bz2 22-Sep-2011 07:40 1.4M
Those are the *sources*.

AFAIK, there are no binary builds outside WinAVR, the "Toolchain"+AS5, and the Linux distros.

They used to be here: http://download.savannah.gnu.org... . They haven't been updated for quite some time, and then somebody of the high esteemed developers purged the whole directory. The link under the official avr-libc homepage (Downloads -> Binary releases) still points here. So much about the open source projects' documentation.

I understand that there's no much reason to maintain the binaries, but on the other hand I don't understand why the person who purged the directory could not place there a simple readme.txt explaining the situation.

westfw wrote:
I dunno. I've built gcc from source; it's a pain in the neck; dependency hell.
I know; I pointed out that that's the place where a lot of work should go. It should be smooth for the user. And the libc should bring in a way lesser hell than gcc.

westfw wrote:
And the size of your AVRStudio5 download isn't going to be dominated by the size of avr-libc, or even the complete WINAVR toolset.
I don't care about Atmel's approach, in the very same way as they don't care about the users' opinion. I am here talking strictly about the open source part of the stuff.

Of the WinAVR toolset I can imagine distribution in various versions, including that with only the core for the power users, which is then comparable in size with the libraries (I am not talking about the Atmel's version, which is clearly incompetently built with lots of redundancy - I pointed that out already several times).

westfw wrote:
I sympathize with the lower-bandwidth problems, but arv-libc looks like the wrong point to address.
If you forget the size issue, wouldn't the potential flexibility of bugfixes justify such approach for you?

And I do appreciate the negative opinions, thanks.

Jan

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

> Again, a few hundreds of megabytes to be downloaded and
> installed doesn't bother you perhaps, given your location,
> but I still find it a waste of resources.

A binary distribution file of the current avr-libc
is 2.5 MiB:

j@uriah 261% ls -l *zip
-rw-r--r--  1 j  staff  2608655 Sep 27 11:19 avr-libc-bin-1.7.2rc2252.zip

I stopped distributing these on savannah though as I've got the impression
nobody has ever been using these binary distributions.

> So much about the open source projects' documentation.

So much about your arrogant attitude I could never stand. Instead of your
rambling here in a forum article, you could have written a bug report, so
this can be fixed. But that would, obviously, remove the source of your
complaint, eventually. I've got the impression you aren't really interested in
that.

Apart from all this, nobody would deny you the right to just download the
sourcecode (or stay in sync with SVN), and compile all that yourself.
It's just that this is not what *most* users would prefer.

Jörg Wunsch

Please don't send me PMs, use email if you want to approach me personally.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

dl8dtl wrote:
> Again, a few hundreds of megabytes to be downloaded and
> installed doesn't bother you perhaps, given your location,
> but I still find it a waste of resources.

A binary distribution file of the current avr-libc
is 2.5 MiB:

j@uriah 261% ls -l *zip
-rw-r--r--  1 j  staff  2608655 Sep 27 11:19 avr-libc-bin-1.7.2rc2252.zip


And how can be explained the difference to the 5-7 megs I reported above, obtained by packing up the directories I think are pertinent to avr-libc from the WinAVR package? Did I make an error in identifying those directories?

dl8dtl wrote:
I stopped distributing these on savannah though as I've got the impression
nobody has ever been using these binary distributions.
This is most probably true and I never said a word against that, contrary.

dl8dtl wrote:
> So much about the open source projects' documentation.

So much about your arrogant attitude I could never stand.

I might've formulated it in a more polite way, and I apologize.

I see you've already removed the link from the homepage. I still think you should place also a readme.txt in http://download.savannah.gnu.org... explaining the situation for those, who come to that place from a different place or a bookmarked link. Shall I open a bug report for that?

dl8dtl wrote:
Apart from all this, nobody would deny you the right to just download the
sourcecode (or stay in sync with SVN), and compile all that yourself.
It's just that this is not what *most* users would prefer.
They don't, because the process as it is now is clumsy. My original premise was, that it would be made user friendly (I of course KNOW that it's not a trivial task and I am not suggesting it to be done now, just want to discuss).

Jan

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

> And how can be explained the difference to the 5-7 megs I reported above,
> obtained by packing up the directories I think are pertinent to avr-libc
> from the WinAVR package?

I didn't try to count it. Anyway, the binary releases I mentioned
never contained the documentation, as it could always be obtained
separately (either as an HTML archive, or as a compressed PDF).
Perhaps that accounts for the difference.

(Apologies accepted)

> I still think you should place also a readme.txt ...

OK, I did.

> My original premise was, that it would be made user friendly

Still, I don't believe there are much more users but you who'd like to
see it that way -- as much as I regret it. I always thought that the
possibility to compile your own is one of the strong points of
opensource software, as it allows to quickly make modifications of
your own (like, own bugfixes, or just tests), and also allows your
debugger to step into the library functions when requested. Being a
FreeBSD user, I'm pretty used to compile really everything myself
(though, of course, most of it in a completely automated way).

But, already when talking to Linux users, most of them think of
compiling something theirselves as a nuisance rather than an
advantage, and when talking to Windows users, they stare at you
already at the idea of being required to have a compiler in the first
place, they see no use in having a tool just for the tools. They want
their setup.exe, everything else appears to be "inconvenient" to them.

Jörg Wunsch

Please don't send me PMs, use email if you want to approach me personally.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

dl8dtl wrote:
> And how can be explained the difference to the 5-7 megs I reported above,
> obtained by packing up the directories I think are pertinent to avr-libc
> from the WinAVR package?

I didn't try to count it. Anyway, the binary releases I mentioned
never contained the documentation, as it could always be obtained
separately (either as an HTML archive, or as a compressed PDF).
Perhaps that accounts for the difference.

Without documentation, it still packs up to 3.3M.
I appended the directories listing, please have a glance, maybe it will ring some bells.

--------

dl8dtl wrote:

> My original premise was, that it would be made user friendly

Still, I don't believe there are much more users but you who'd like to
see it that way


And what if they don't *see* it at all? What if I teach avr-gcc (the "driver" program) to build the required parts of libraries upon first use? I believe the Windows users are used to a couple of annoying long-lasting operations per day, and this one could even be properly documented.

dl8dtl wrote:
I always thought that the
possibility to compile your own is one of the strong points of
opensource software, as it allows to quickly make modifications of
your own (like, own bugfixes, or just tests)
That is one of the two key reasons for this discussion.

dl8dtl wrote:
, and also allows your
debugger to step into the library functions when requested.
I could allow to go for that through a switch.

dl8dtl wrote:
Being a
FreeBSD user, I'm pretty used to compile really everything myself
(though, of course, most of it in a completely automated way).

But, already when talking to Linux users, most of them think of
compiling something theirselves as a nuisance rather than an
advantage,

That's because the "automation" you mentioned is too fragile and too often broken in the modern Linuces. I speak from experience of two decades of casual user=level use of (fight with) various Unices/Linuces. I know nothing about FreeBSD, MacOSX or other flavours of UNIX-like OS.

dl8dtl wrote:
and when talking to Windows users, they stare at you
already at the idea of being required to have a compiler in the first
place, they see no use in having a tool just for the tools. They want
their setup.exe, everything else appears to be "inconvenient" to them.
True, but I believe that this is not typical for the majority of avr-gcc users, given their specific interest. Moreover, if you make it hidden enough, it should satisfy the rest.

Thanks for your comments.

Jan

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

I always thought that the
possibility to compile your own is one of the strong points of
opensource software, as it allows to quickly make modifications of
your own (like, own bugfixes, or just tests), and also allows your
debugger to step into the library functions when requested.

That is a strong point. Another equally strong point, for me at least, would be the ease of just acquiring a binary release. My dream re this (for both avr-gcc and avr-libc) would be identical paralell releases for "the dominating platforms". I.e. avr-gcc with avrlibc released pre-built for both Wondoze and e.g. as a .deb package based the exact same sources etc..

Optimizing "a few" megabytes, be it for faster downloads or disk footprint, would be a waste IMO. If the argument for this optimization is merely "it's waste, no matter how small" then this is optimization for optimizations own sake, or "just for the principle". Meaningless, if you ask me.

Or: Any overall optimization will always lead to local un-optimizations. Peephole optimization just for the principle is sub-optimization.

What download times do you have, wek? How big is your hard drive?

I realize that far from all have e.g. > 1 MbPS, but I just did a test d/l of WinAVR from work and it took 30 seconds flat. (And it comes with the avrlibc documentation baked in.) I'd have to look long and far to find a disk that is smaller than 120 gig.

Why should any of the honourable people wokring on avr-gcc and/or avrlibc spend time on this relatively meaningless optimization, when there most likely is an abundance of more pressing things to fix?

As of January 15, 2018, Site fix-up work has begun! Now do your part and report any bugs or deficiencies here

No guarantees, but if we don't report problems they won't get much of  a chance to be fixed! Details/discussions at link given just above.

 

"Some questions have no answers."[C Baird] "There comes a point where the spoon-feeding has to stop and the independent thinking has to start." [C Lawson] "There are always ways to disagree, without being disagreeable."[E Weddington] "Words represent concepts. Use the wrong words, communicate the wrong concept." [J Morin] "Persistence only goes so far if you set yourself up for failure." [Kartman]

Last Edited: Tue. Sep 27, 2011 - 03:12 PM
  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Support here is difficult enough with a handful of WinAVRs and a couple of toolchains each with a known copy of AVR-LibC. Imagine if everyone were using a (possibly) locally patched AVR-LibC - then trying to sort the "compiler bugs" wheat from all the chaff could become a right old game! 3 days of "try this".. "what about that?" .. then you finally discover they hacked gcrt1.S for something they needed on the last project and had forgotten about it.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:
Quote:
Straight from the project at savannah:
avr-libc-1.7.2rc2252.tar.bz2 22-Sep-2011 07:40 1.4M

Those are the *sources*.

Oops. So they are. Blush. I was tricked by the "deep" directory structure, which I didn't think got created till build time.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

dl8dtl wrote:
> I still think you should place also a readme.txt ...

OK, I did.

I can see it in the directory listing on http://download.savannah.gnu.org... linking to http://download.savannah.gnu.org... , but when clicking on it it will redirect me through 302 to http://download.savannah.gnu.org... which in turn redirects me again through 302 to http://mirror.lihnidos.org/GNU/s... and that throws a 404 Not Found...

Shouldn't I submit that bug report anyway?

Jan

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

JohanEkdahl wrote:
Quote:

I always thought that the
possibility to compile your own is one of the strong points of
opensource software, as it allows to quickly make modifications of
your own (like, own bugfixes, or just tests), and also allows your
debugger to step into the library functions when requested.

That is a strong point. Another equally strong point, for me at least, would be the ease of just acquiring a binary release.

Would you be concerned about acquiring pre-built binaries, if I promise they would appear out of the blue sky (through automated compilation controlled directly by avr-gcc) after the first attempt to compile (link) for a given avr model?

JohanEkdahl wrote:
Optimizing "a few" megabytes, be it for faster downloads or disk footprint, would be a waste IMO.
Would it be a waste if it would come as a byproduct to the distribution system described above? It would come at the cost of wasted PC cycles at the compilation time, and potentially wasted disk space if built for many avr models, anyway.

clawson wrote:
Support here is difficult enough with a handful of WinAVRs and a couple of toolchains each with a known copy of AVR-LibC. Imagine if everyone were using a (possibly) locally patched AVR-LibC - then trying to sort the "compiler bugs" wheat from all the chaff could become a right old game! 3 days of "try this".. "what about that?" .. then you finally discover they hacked gcrt1.S for something they needed on the last project and had forgotten about it.
This is the argument I was most afraid of (plus the unknowns yet to come).

I could start to babble about checksumming (or, if you want a fancier expression, hashing) at this place; and/or a flag to force a "known good" state (plus appropriate documentation and education). Could this be the direction towards an acceptable answer?

Thanks,

Jan

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Stuffing that kind of scheme into GCC would probably hit a lot of resistance,
as the compiler folks wouldn't consider it being their job to do "unusual
things under the hood". Arranging for that kind of things is the job of
the make utility (or comparable systems).

Regarding the readme file on savannah, you'll either have to wait for the
indicated mirror to pick up the file, or you have to follow the instructions
there about how to get an unmirrored version (you have to manually edit the
URL for that). No need for a bug report, delays of up to 24 h in the mirroring
process are pretty normal.

Sorry, I already deleted my binary .zip file, so I can't tell you about any
differences. However, the script to create that binary distribution you can
find in the devtools/ directory, if you've already got the SVN tree. That
way, you can just see yourself what it does, and what's included.

Jörg Wunsch

Please don't send me PMs, use email if you want to approach me personally.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

dl8dtl wrote:
But, already when talking to Linux users, most of them think of
compiling something theirselves as a nuisance rather than an
advantage, and when talking to Windows users, they stare at you
already at the idea of being required to have a compiler in the first
place, they see no use in having a tool just for the tools. They want
their setup.exe, everything else appears to be "inconvenient" to them.
I've occasionally compiled open source code myself,
but it's almost always been a pain.
The only exceptions I can think of were X (!), GLPK and lp_solve,
all at least several years ago.
IIRC the X build system was a horror.
Had it not gone well, I had no chance of fixing it.
The excretions of autoconf aren't very friendly either.

After a long hard struggle,
typing "make install" as root is something to inspire fear and trepidation.
With bingo's scripts, I managed to sidestep the problem.
I made the top-level target directory owned by me and changed it back after installation.

@wek, bingo: do the bingo scripts work under cygwin?

Despite all that, I'm about to start a job that I expect will require me to do a lot more compiling.
Would source RPMs, when available, provide me with good models for compiling from source?
E.g. to compile boss's version of fractious 7.2,
get source RPM for fractious 6.66, replace its source with the boss's and type make a lot.
Note that I've read about source RPMs, but haven't actually looked at one yet

"SCSI is NOT magic. There are *fundamental technical
reasons* why it is necessary to sacrifice a young
goat to your SCSI chain now and then." -- John Woods

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

IIRC the X build system was a horror.

[OT]I was trying to use a pre-release version of the Intel GMA3500 graphics adapter and Linux for a new board that Intel provided us at one stage and because it was "bleeding edge" it only matched the HEAD version of the X in the repository (in fact I think it was worse and it was a development branch) so I had to build X to match. It was without doubt one of the worst open source building fiascos I have ever endured (for several weeks). There were endless build errors I had to resolve in turn (partly because of version inter-dependencies) and I'd happily have those few weeks of my life back if I could! (I suppose there was a sense of satisfaction at the end but sadly rather let down by the fact that it turned out that the video BIOS in the onboard graphic adapter was also pre-release and would not work with the latest driver code)

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

dl8dtl wrote:
Stuffing that kind of scheme into GCC would probably hit a lot of resistance,
as the compiler folks wouldn't consider it being their job to do "unusual
things under the hood". Arranging for that kind of things is the job of
the make utility (or comparable systems).
Make (or comparable - here, AVRStudio(s)) is already taken by the users.

But will the gcc folks stop me to stuff it into a program called "avr-gcc" in my own distribution of the "toolchain"? Is there anything in the licences preventing me to do that? Or would Atmel's lawyers step in in protection of the three-letter acronym which they claim is not an acronym?

I don't think this could stop me.

(Don't forget, I am NOT going to go for nor push this thing - just want to discuss it).

dl8dtl wrote:
Regarding the readme file on savannah, you'll either have to wait for the
indicated mirror to pick up the file, or you have to follow the instructions
there about how to get an unmirrored version (you have to manually edit the
URL for that). No need for a bug report, delays of up to 24 h in the mirroring
process are pretty normal.
Ah I see, thanks. I have the bad habit to neglect the fineprint :-)

dl8dtl wrote:
Sorry, I already deleted my binary .zip file, so I can't tell you about any
differences. However, the script to create that binary distribution you can
find in the devtools/ directory, if you've already got the SVN tree. That
way, you can just see yourself what it does, and what's included.
Now that's the prime example of why self-building things under UNIX-like systems can be a pain (on the easy side, I did not svn but I have simply downloaded the source pack). I guessed you are talking about devtools/make-binary-dist.sh (right?). Then, after a bit of experimenting, I guessed you meant to run it from the root of the sources, i.e. not from the devtools subdirectory (right?). Then, configure exercised all those obscure scripting languages it discovered, for some ten minutes. Then things went simply wrong (see attachment)... :-)

Jan

Attachment(s): 

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

If you are concerned about the size the built library occupies, why do you want to build it per-device?

Library built per-core (avr2, avr31, ...) is smaller because each multilib version covers many derivatives, despite a derivative-wise build.

Did you download the avr-libc sources and try to change them for your needs? The crt is built on a device level, so you can try an adapt it to other objects/libs you want to have per device and not per core.

Did you build it without debug information and stripped the objects? Should save some bytes, that.

avr-libc's structure reflects the multilib structure of avr-gcc. I don't hink there is much use for an "official" avr-libc that build itself fine grained because it does not fit to avr-gcc's multilib layout.

Or you could adapt avr-libc so that it depicts it's multilib layout directly from (avr-)gcc, i.e. similar to newlib build process, and provide a patch for that. All you then have to do to change the multilib description in avr-gcc and avr-gcc and avr-libc still integrate seamless.

As you know, avr-libc is an open project and feature is implemented by the one that cares the most — which appears to be you.

avrfreaks does not support Opera. Profile inactive.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

wek wrote:
Then things went simply wrong (see attachment)... :-)

[png]

That's just a part of the log message.

Instead of posting it as png image you should make a flash (or mp4 or whatever) animation and post it here so that everyone can see the messages passing by.

avrfreaks does not support Opera. Profile inactive.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

> Instead of posting it as png image you should make a flash (or
> mp4 or whatever) animation and post it here so that everyone
> can see the messages passing by.

Hey, you're kidding.

Instead:

./devtools/make-binary-dist.sh > log.txt 2>&1

and post log.txt as an attachment.

> But will the gcc folks stop me to stuff it into a program called "avr-gcc"
> in my own distribution of the "toolchain"?

Certainly not, but you'll very likely confuse people.

> Now that's the prime example of why self-building things under UNIX-like
> systems can be a pain

Here, you err. All is working smooth under a Unix-like system (I wrote the
script to ensure those binary releases were created exactly the same way for
each release), it's just that there is no decent support for that under Windows.

> Then, after a bit of experimenting, I guessed you meant to run it from the
> root of the sources, i.e. not from the devtools subdirectory (right?).

Well, it's a developer tool, not an end-user tool. Developers are expected to
at least have a glance at it ... there, you'll see:

# Build a binary distribution archive.
#
# Assumes a bootstrapped source tree to be present in ${topdir}.

...
topdir=${1:-.}
...

That construct means: "If the script is run with an argument, use that as
the top-level directory, otherwise use `.' (the current directory) as
the top-level." So, either do it as you did, or if you want to run it from
within the devtools/ directory, run it as

./make-binary-dist.sh ..

Jörg Wunsch

Please don't send me PMs, use email if you want to approach me personally.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I'we seen stuff like wek's pict. happen (loops .. if that is what it is) , if there were symlinks in the source.
Even Cygwin isn't happy with all symlinks.

But EW does compile it.

@wek

Do you know that the binaries from avr-libc is the same under linux and windows.
You could build it under Linux , and copy the files to windows.
You'd just need to translate \n to \r\n in the .h files.
Or ftp them with an ascii option

/Bingo

/Bingo

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Bingo600 wrote:
Do you know that the binaries from avr-libc is the same under linux and windows. You could build it under Linux, and copy the files to windows.
You'd just need to translate \n to \r\n in the .h files.
The 4.6.1 Toolchain for Win32 I posted here some weeks ago is built completely under Linux and it runs fine under windows. Same for binutils and avr-libc; it's all built under linux and just cross compiled to mingw.

All plain vanilla. No patching of headers etc.

Never tried to build a windows toolchain under windows, it's too painful. And cygwin is not a small package. Just building from a dos promt (cmd.exe) -- is that supposed to work at all?

avrfreaks does not support Opera. Profile inactive.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

SprinterSB wrote:
If you are concerned about the size the built library occupies, why do you want to build it per-device?
My concern was internet bandwidth rather than disk space. But even that - most users use only one AVR model, some use maybe two or three - so they would benefit from not having a dozen libraries they don't need at all.

Of course your comments about compromise solutions still apply - it's just that it's not necessary to have (build) the libraries for all the cores, only to those which are in use.

SprinterSB wrote:
As you know, avr-libc is an open project and feature is implemented by the one that cares the most — which appears to be you.
I repeat, I am just discussing ideas, for fun, rather than for any well-formed purpose.
----

The failed build: thanks for the offered help, but don't worry, I'll manage (or drop it, it's not THAT important). That sequence simply repeated itself and the number increased permanently; and Task Manager witnessed that new and newer instances of make were run. At around 200 it simply started to choke the PC so I had to kill it (it all took maybe a minute or two; I repeated the whole fun just to take that screenshot). Make clean and make distclean resulted in the same behaviour. Running ./configure or said script again resulted in the process stuck at something like "testing if make sets {MAKE}" or whatever, with the population of makes exploding again... :-)

This all was run under sh from the WinAVR distribution; running sh from msys/MinGW appears to result in a more civil behaviour. If time permits I'll try run the rest on background tomorrow (on foreground, a firmware for a chip is being worked on, chip based on a RISCish 8-bit core, no C compiler just a good old fashioned assembler, a job truly to my liking... ;-) ).
----

Joerg wrote:

Well, it's a developer tool, not an end-user tool. Developers are expected to
at least have a glance at it ...
I am a user (in Win/Lin) and I never claimed otherwise. And we are talking about user experience.

I don't complain, I just (ab)used it as an example - almost all *nix source package requires this level of expertize. I hope you don't mind.

----
Thanks all for your comments.

Jan

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

wek wrote:
SprinterSB wrote:
If you are concerned about the size the built library occupies, why do you want to build it per-device?
My concern was internet bandwidth rather than disk space. But even that - most users use only one AVR model, some use maybe two or three - so they would benefit from not having a dozen libraries they don't need at all.
My laptop is pretty old; 10 years now. And it has to problems with the toolchains. There are other pieces of software that eat up much more. Some WinAVR distributions come with AVR32 support; cou can throw that away — of course only after downloading.

But building the tools from source won't save you bandwiths: just build avr-gcc and you never complain again that WinAVR it too big ;-)

wek wrote:
Of course your comments about compromise solutions still apply - it's just that it's not necessary to have (build) the libraries for all the cores, only to those which are in use.
Would be a good exercise in make.

Don't know if there is software that uses that approach. TeX + metafont do that way, i.e. just compile a font from source if it's actually needed.

Quote:
SprinterSB wrote:
As you know, avr-libc is an open project and feature is implemented by the one that cares the most — which appears to be you.
The failed build: [...]. That sequence simply repeated itself and the number increased permanently; and Task Manager witnessed that new and newer instances of make were run. At around 200 it simply started to choke the PC so I had to kill it (it all took maybe a minute or two; I repeated the whole fun just to take that screenshot). Make clean and make distclean resulted in the same behaviour. Running ./configure or said script again resulted in the process stuck at something like "testing if make sets {MAKE}" or whatever, with the population of makes exploding again... :-)
That's nothing avr-libc is to be blamed for, it's the build environment. I observed similar problems in project where I tried to use sub-make. No go, make get's no parameter and as "all" is commonly the first target, i.e. default, you see make calling itself again and again and again...
Quote:
This all was run under sh from the WinAVR distribution;
If you like to build, you need a proper build environment. Examples are cygwin and linux.

avrfreaks does not support Opera. Profile inactive.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

> I am a user (in Win/Lin) and I never claimed otherwise.

Not really. As a user, you'd simply ask for your setup.exe.

Sorry, you sounded as if you were about to start becoming a developer
(since you were talking about making your own distribution, where a
command named "avr-gcc" would do some fancy stuff). As a mere user, I
wouldn't have recommended you that script at all.

> almost all *nix source package requires this level of expertize.

Well, if you call it "expertize". In contrast, to me, running/maintaining
Windows would require a lot more of "expertize", just to keep it going in a
reasonable manner. If you don't mind. ;-)

As I don't want to spend any much time into that kind of expertize, I
simply resort to environments where all that kind of development work
is just going fine, straight from the beginning. No, I'm not
complaining, rather *explaining*, why things are the way they are. To
me (and certainly not only to me), developing software under Windows
would just quickly drive me away, while developing it under a unixoid
system lets me concentrate on the software itself, rather than the
operating system. If you look around, a large percentage of all
opensource software is being developed under unixoid systems. There
must be a reason …

Jörg Wunsch

Please don't send me PMs, use email if you want to approach me personally.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

SprinterSB wrote:
I wrote:
it's just that it's not necessary to have (build) the libraries for all the cores, only to those which are in use.

Would be a good exercise in make.
Would be a good exercise how to *avoid* make (and the rest of that stuff). Running make on already built avr-libc tree (i.e. just checking dependencies, nothing to compile) took now around half an hour. Not a top-notch machine but a fair one; nothing else relevant running on it; one core 100% all the time, with almost no disk activity. I'll try to report tomorrow about how long a clean make took.

SprinterSB wrote:
I wrote:
... the failed build...
That's nothing avr-libc is to be blamed for, it's the build environment.
Isn't the configure script, together with the three obscure scripting interpreters it uses, supposed to check appropriateness of that build environment???

SprinterSB wrote:
...cygwin...
That was the first to try. Failed - no avr-gcc, apparently... :-)

---

dl8dtl wrote:
> I am a user (in Win/Lin) and I never claimed otherwise.

Not really. As a user, you'd simply ask for your setup.exe.

Sorry, you sounded as if you were about to start becoming a developer
(since you were talking about making your own distribution, where a
command named "avr-gcc" would do some fancy stuff).


In almost every post I made in this thread I stressed that I am not going to DO that thing, just want to discuss. Isn't that "userish" enough? :-)

dl8dtl wrote:
> almost all *nix source package requires this level of expertize.

Well, if you call it "expertize". In contrast, to me, running/maintaining
Windows would require a lot more of "expertize", just to keep it going in a
reasonable manner. If you don't mind. ;-)

I now deleted what started to be a really lengthy answer.

I both agree and disagree to a certain extent, but I really would like to have the opportunity to discuss this (and the rest of your post) with you elsewhere and under less pressure.

Jan

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Quote:

Would you be concerned about acquiring pre-built binaries, if I promise they would appear out of the blue sky (through automated compilation controlled directly by avr-gcc) after the first attempt to compile (link) for a given avr model?

Would&could I be sure they stand on the same code base as the avrlibc that appeared out of the blue sky three months earlier after my first attempt to build for another AVR model?

Quote:

potentially wasted disk space if built for many avr models, anyway.

I note "potentially", which is a understatement at best.

As of January 15, 2018, Site fix-up work has begun! Now do your part and report any bugs or deficiencies here

No guarantees, but if we don't report problems they won't get much of  a chance to be fixed! Details/discussions at link given just above.

 

"Some questions have no answers."[C Baird] "There comes a point where the spoon-feeding has to stop and the independent thinking has to start." [C Lawson] "There are always ways to disagree, without being disagreeable."[E Weddington] "Words represent concepts. Use the wrong words, communicate the wrong concept." [J Morin] "Persistence only goes so far if you set yourself up for failure." [Kartman]

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

Well, if we're "discussing", I think that ultimately any improvements aimed primarily at conserving disk space are a wasted effort that would have been better spent on something else. Disks are cheap and huge; a terrabyte fits in your shirt pocket...

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

wek wrote:
Running make on already built avr-libc tree (i.e. just checking dependencies, nothing to compile) took now around half an hour. Not a top-notch machine but a fair one; nothing else relevant running on it; one core 100% all the time, with almost no disk activity. I'll try to report tomorrow about how long a clean make took.
Jan

@wek
As you mention , the machine can do something about it.

I can build/make a complete avr-toolchain in 5min41sec , on my new pc, using my buildscript modified for "make -j8".
Its a Core-I7 Quad with 8G Ram , and with Western Digital Greenline (4K Blocks/Physical Sectorsize) 1.5T disk solely for Ubuntu.

At first it was 17min , and i was seriously disapointed , as i got it to speed up the toolchain building.

Then i read that the disk was lousy if your partition didn't start on a Physical Block boundry (for me 4k). Due to some old heritage from DOS , most partitions starts on sector 63, witch is not 4K alligned. And Ubuntu (pre 10.x) didnt take this into consideration (nor does Xp).

I scratched my disk , realigned my extfs3 partition (manual fdisk) to start on sector 64 (4K boundry). And got 5min41sec.

I wonder what the speed would be if i use a ramdisk.
Maybe not that much more , as it seems that most of the 8G ram , when unused. Is being allocated to the kernel , and used as diskcache.

But as SprinterSB says : avrlibc diskusage is "peanuts" compared to build the full TC. My prefix (output dir) , containing the extracted sources , objects and resulting TC , takes up 866MB.

You can free most of that , by deleting the source & build dirs , but then you loose the sourcelevel debugging of avr-libc , and compiler internals.

Ooopzz quite OT :oops:

I got carried away ...

Back on topic

I was under the impression that you would like the "Per MCU" building of avrlibc , because it would solve the problem with the "Far access" , as the flashsize could be specified per MCU , and not a general "per family" thing.

Is the disk/download usage subject , a "trick" to get to the above. :wink: :wink:

/Bingo

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

> Would be a good exercise how to *avoid* make (and the rest of that stuff).
> Running make on already built avr-libc tree (i.e. just checking dependencies,
> nothing to compile) took now around half an hour. Not a top-notch machine but
> a fair one; nothing else relevant running on it; one core 100% all the time,
> with almost no disk activity.

What makes you think that avoiding make would solve *anything* here?

No, it's not make itself that causes you the trouble. Something's wrong
with your build environment, in some way, and that won't be fixed by
hacking the functionality of make into avr-gcc itself. The Unix approach
is: one tool per job, not one big hammer that covers everything. Yes,
make is pretty old now, and there are also different tools around (cmake
or scons, for example), but if your build itself is so damn slow, they
won't save your back either.

> In almost every post I made in this thread I stressed that I am not
> going to DO that thing, just want to discuss.

Who, do you think, would be eventually doing it?

It's IMHO a waste of time to discuss things that nobody is willing to
implement, ever. So far, you are completely alone with your opinion,
so why do you think would anyone else implement your thoughts?

If it's not you, it won't happen, I'm sure.

Btw., my avr-libc build run just completed:

j@uriah 303% time gmake
...
93.990u 97.462s 3:34.58 89.2%   2629+601k 5677+310io 94pf+0w

I've been deliberately *not* using »make -j2« to get both of my
CPU cores involved, so this is a single-CPU run, and it completes
in 3.5 minutes. It uses a decent filesystem on decent disks
though, which might contribute a bit to the speed. But then, the
CPU usage was about 3 out of those 3.5 minutes, so it's at least
not primarily disk-bound anyway. The CPU is a:

CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (1989.88-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x40f32  Family = f  Model = 43  Stepping = 2
  Features=0x178bfbff
  Features2=0x2001
  AMD Features=0xea500800
  AMD Features2=0x1f

several years old now, so anything but "brand new".

Jörg Wunsch

Please don't send me PMs, use email if you want to approach me personally.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

I see mention of CygWin above - just to note that a few years back I had a PM exchange with Eric when I was trying to cross compile a Z80 C compiler and he told me that the majority of WinAVR is built with Msys/MinGW (and then helped me set up a similar environment). So I'd recommend setting that up if you plan to try an do the building in a Windows host.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

My workplan was ruined today again, as more and more unexpected and unpleasant jobs surfaced during the day... So sorry, but just some quick answers, unordered:

  • as I said above I the option which finally worked was Msys/MinGW (although I suspect the stuff coming with WinAVR is the same family :-) )
  • make -r resulted in the "do nothing" make of a few tens of seconds (compared to half an hour for the plain make) and the "real" make (after clean) lasting around 5 minutes - yes the implicit rules are one of the reasons why I don't like make (understandably, it takes time to check absence of zillions of made-up files, every time walking through layers from msys's sh down to Windows's file system)
  • I couldn't persuade the script through a shell variable MAKE to use make -r, which shows where in the user/developer status I am; so the build (involving two make "recursives", first for all and second for install) took around a hour
  • first I made the mistake to use my trusty 2007 vintage WinAVR (which is default in PATH), which resulted in all xmegas' libraries missing
  • then I used the 2010 vintage, that resulted in the figure Joerg gave above (around 2.5meg zip); the major difference to the WinAVR20100110 sizes I reported above are due to missing doc, missing /avr/lib/avrxmega3 directory, missing /avr/lib/ldscripts, and missing libobj.a files all around
  • > It's IMHO a waste of time to discuss things > that nobody is willing to implement, ever.
    I have learned a couple of lessons, so from my selfish point of view it's not a waste. Forget about avr-libc for a while and consider for example a computer-based system deployed in dozens in the wild, with expensive data connectivity (dialup, GPRS), with software/firmware custom built (and rebuilt occasionally) for each site, being configured to a particular hardware or particular user requirement on that site. Now I know that bugfixes and upgrades through sources is viable, but I have to checksum (fingerprint, hash, whatever fancy name) last built sources; store original sources and have a plan to revert back to them; and use only trusty known good tools, strictly rejecting replacements.
Thanks all.

Jan

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

> I have learned a couple of lessons, so from my selfish point of view it's not a waste.

Agreed.

> every time walking through layers from msys's sh down to Windows's file system

There are no fat layers between, and when "make" is running, it doesn't invoke a shell
to check for a file. Msys/MinGW is really not much of a layer (unlike Cygwin), the only
thing the Msys/MinGW layer adds is to convert a name like "/c/foo/bar" into
"C:/foo/bar", before passing it on to the Win32 API. In all other respects, MinGW
just exposes the Win32 API only, not adding any kind of emulation layer.

If already just *probing* for a file (and its associated timestamp) takes (too much) time,
then make is the most lightweight of the tools in question. Other tools (like
scons) perform a checksumming (hash) on the files in order to determine whether
the file has changed, so they not only have to find the file in the first place
(like make) but they also have to *completely read* it in order to decide whether
its dependencies are to be rebuilt or not.

I don't really understand why you'd like to avoid make's builtin rules though
(which is the effect of »make -r«), you lost me on that.

Jörg Wunsch

Please don't send me PMs, use email if you want to approach me personally.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

wek wrote:
...cygwin... That was the first to try. Failed - no avr-gcc, apparently... :-)
Are you kidding? It doesn't come with gcc pre-built for each imagineable combination of configure options.

• get binutils — configure it — build it — install it
• get gcc — configure it — build it — install it
• get avr-libc — configure it — build it — install it

That's it. Straight forward.

wek wrote:
I am just discussing ideas, for fun, rather than for any well-formed purpose.
Ah, got it now. You are just bored...

avrfreaks does not support Opera. Profile inactive.

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

SprinterSB wrote:
wek wrote:
...cygwin... That was the first to try. Failed - no avr-gcc, apparently... :-)
Are you kidding? It doesn't come with gcc pre-built for each imagineable combination of configure options.

• get binutils — configure it — build it — install it
• get gcc — configure it — build it — install it
• get avr-libc — configure it — build it — install it

That's it. Straight forward.

Straight to more problems forward.

I can of course run avr-gcc and kin from within cygwin's shell (and the PATH shell variable is set appropriately); it's the configure script (and its three obscure scripting interpreters) who couldn't find it. I then simply chose the least resistance path through msys/minGW.

SprinterSB wrote:
wek wrote:
I am just discussing ideas, for fun, rather than for any well-formed purpose.
Ah, got it now. You are just bored...
Yes, if you call my attempt to escape from unexpected loads of intellectually unchallenging and hateful work boredom, then yes.

On the other hand, I don't quite understand the "ignore it unless you see direct benefit from it" approach I see here so often. Must be some cultural difference.

Jan

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

dl8dtl wrote:
> every time walking through layers from msys's sh down to Windows's file system

There are no fat layers between, and when "make" is running, it doesn't invoke a shell
to check for a file. Msys/MinGW is really not much of a layer (unlike Cygwin), the only
thing the Msys/MinGW layer adds is to convert a name like "/c/foo/bar" into
"C:/foo/bar", before passing it on to the Win32 API. In all other respects, MinGW
just exposes the Win32 API only, not adding any kind of emulation layer.

Then it's the winapi which is inefficient, or whatever layer(s) are below that. I am not going to investigate, as I cut down the number of requests avoiding the implicit rules of make.

dl8dtl wrote:
I don't really understand why you'd like to avoid make's builtin rules though
(which is the effect of »make -r«), you lost me on that.
To cut down the number of requests to the filesystem through the inneficient winapi, see above.

Jan

  • 1
  • 2
  • 3
  • 4
  • 5
Total votes: 0

wek wrote:
SprinterSB wrote:
• get binutils — configure it — build it — install it
• get gcc — configure it — build it — install it
• get avr-libc — configure it — build it — install it

That's it. Straight forward.

Straight to more problems forward.
If you like to play with the tools you'll have to be brave and face the challenge of doing the things that you will have to do anyway when you want to get started with the tools from inside.

Before tweaking/changing/extending stuff, the first step is to do a plain vanilla build as recommended, and only after that to start deviating in whatever direction — other build environment, configure script, etc. — and look if you are fine with it and it works for you.

For anyone who's just starting all that seems unnecessary ballast, but a shortcut too early is likely to cause frustration and waste of time instead of being fast and smart.

avr-gcc and avr-libc are not completely independent of each other; they are loosely coupled and avr-libc relies on features of avr-gcc. If avr-gcc does not work as intended, you are stuck: PR44643 and PR45261 are two examples.

wek wrote:
if you call my attempt to escape from unexpected loads of intellectually unchallenging and hateful work boredom, then yes.

On the other hand, I don't quite understand the "ignore it unless you see direct benefit from it" approach I see here so often. Must be some cultural difference.

I don't think "learning something" is "no direct benefit". IMO it's more direct benefit than getting something to work without knowing why it works or at least to have a slight impression why it works or does not work.

It just was hard to grasp what you want. And I got the impression that you focus more on what does not work (make is scrap, configure is scrap, etc.) instead of trying to get in medias res.

Of course do make and configure and auto-tools and gcc have shortcomings and invent legacy patterns and historically grown structures. Anyone who is working with that tools is aware of that.

avrfreaks does not support Opera. Profile inactive.