Apache NetBeansHow to store external libraries in the NetBeans Hg repository
In the spirit of building on the shoulders of giants, NetBeans takes advantage of external libraries which are not developed on netbeans.org. Those libraries are either open-source software, or binary-only software but with liberal licenses. A few examples: Apache Tomcat; JUnit; JavaHelp (runtime); javac compiler; JSR-88 interface classes.
For convenience, these libraries are stored
in the same Hg repository as the source code under CDDL/GPL.
They are placed in well-known places in the source tree.
The license text is associated with the binary file
to make it clear which terms and conditions the users/developers must agree to
besides being compliant with the CDDL/GPL itself.
Only source code covered by the CDDL/GPL (or BSD, in the case of samples)
can be hosted in the http://hg.netbeans.org/main/ repository.
As the NetBeans Hg tree is growing, we need to initiate stricter rules
and check that all external binary files have a correct associated license.
There are also several recommendations
on avoiding unnecessary additions of binary files into Hg.
The build system will automatically check if all binary files under <nbmodule>/external
are stored correctly with appropriate license and all required information.
<nbmodule> means NetBeans project module, e.g. external is on same level as nbproject.
Failing to do so will result in a broken build!
Questions:
-
I need to store some binaries in my own VCS repository. Should I follow same rules as well? No, you do not have to. You can store your binaries under release/modules/ext/, more details are described in
harness/README -
My binary is not a library and I need to store it somewhere else. It has been also created under CDDL. Then you should update
nbbuild/antsrc/org/netbeans/nbbuild/extlibs/ignored-binaries -
How can I check all is all right? Run
ant verify-libs-and-licenses
Here are the rules NetBeans committers must follow when placing external libraries into NetBeans Hg:
-
Legal due diligence must be observed before using a new external library, to make sure that the library license is suitable for use in NetBeans.
-
All external binaries should be stored under a subdirectory named
<nbmodule>/external, and nowhere else. (For thecontribrepository, the path will becontrib/<nbmodule>/external.) -
External binaries are versioned in Hg. ExternalBinaries describes how the actual binary content is stored outside Hg, while the Hg repository actually tracks the SHA-1 hash of the binary.
ant download-selected-extbinssuffices to download all external binaries in a fresh checkout. -
Each external binary should have a corresponding license file stored in the same directory as the binary itself. You will upload the binary itself through the Web form, but will add the license file directly to Mercurial (e.g.
hg add external/somelib-x.y.z-license.txt). -
The name of the binary must follow the convention
somelib-x.y.z.jarorsomelib-x.y.z.zipwherex.y.zis the version number. The corresponding license file must be namedsomelib-x.y.z-license.txt. -
All license files should be in UTF-8 encoding with appropriate line and paragraph breaks. The license file must end with a newline. Lines should not exceed 80 characters.
-
The license file should follow a specific format. Details below.
License file format
License files should be in the following format:
Name: SomeLib
Version: 1.2.3
Description: Library for management of some blah blah blah.
License: Apache_V20 [SeeNoteRegardingNormalizedNames]
OSR: 1234 [OSRNumber,ReferToLFIPreviously;SunInternalLegal]
Origin: http://www.xyz.org [WhereFile(s)WereDownloadedFrom]
Files: xyz.jar, xyz-doc.zip, xyz-src.zip [Optional;SeeBelowForExplanation]
Source: URL to source [MandatoryForLGPL,OtherwiseOptional]
Comment: needed until NB runs on JDK 6+ [Optional:WhyIsThisLibraryHere]
Use of SomeLib version 1.2.3 is governed by the terms of the license below:
[TEXTOFTHELICENSE]As hinted at above, the OSR field refers to a Sun-internal system. Those contributing patches from outside of Sun can leave this field blank. Also note that a single license file may cover multiple JAR files from the same project. For example, if your patch depends on a third-party library distributed under the same license as two JARs, you will only need one license file and can account for both of these JARs in its Files header.
If the Files header is not present, then a license name-x.y.z-license.txt must
correspond to a binary name-x.y.z.jar or name-x.y.z.zip.
If present, it should list the names of all binaries to which it corresponds.
The header fields are read during the build process and removed. Therefore this information will not appear in the final build or NBMs.
Template-based licenses
If there is template-based license (like BSD one http://www.opensource.org/licenses/bsd-license.php),
e.g. the license file has several ad hoc places to be updated accordingly.
The template itself should have the license file stored under nbbuild/licenses
with well-defined tags <pre>*TAGNAME*</pre>; these tags will be replaced during the build.
Template-based licenses stored along with the binary in Hg
must have be in original form as they came with binary:
Example BSD License, as it is stored in nbbuild/licenses:
Copyright (c) '''YEAR''', '''OWNER'''
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of '''ORGANIZATION''' nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.Example BSD License, as it is stored in Hg along with binary:
Copyright (c) 2007, NetBeans
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of NetBeans nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.NBM build, managing correct license for NBMs
Required licenses should be listed in project.properties.
(There still must be a license along with the binary in Hg.)
The new entry will be called extra.license.files,
where the license files will be relative to project basedir,
e.g.
extra.license.files=external/x-1.0-license.txt,external/y-2.0-license.txtThis will create an NBM with two extra licenses besides the usual CDDL. This also maintains compatibility with the current build system.
As a convenient shortcut for the common case that you simply want to
copy some files to the target cluster
(but cannot use the release directory since third-party binaries are involved),
you may use the newly introduced release.* Ant properties
which should be specified in project.properties.
Each key names a file in the source project;
the value is a path in the target cluster.
Any such pair will automatically:
-
Copy the source file to the cluster. (No need to override the
releaseAnt target.) -
Cause the target file to be included in the NBM file list. (No need to add to
extra.module.files.) -
In the case of
release.external/*properties, cause the associated binary to be included in the NBM license. (No need to override thenbmAnt target or add toextra.license.files.)
Example (from the form module):
release.external/beansbinding-0.6.1.jar=modules/ext/beansbinding-0.6.1.jar
release.external/beansbinding-0.6.1-doc.zip=docs/beansbinding-0.6.1-doc.zip(Note: if you wish for the binary to be in the classpath of the module as a library,
you will still need a <class-path-extension> in your project.xml.)
You can also use a ZIP entry on the left side and it will be extracted from the ZIP
to your cluster:
release.external/stuff-1.0.zip!/stuff.jar=modules/ext/stuff-1.0.jarNormalized names
There will be a license repository under nbbuild/licenses
where all licenses in use should be available.
Each license type will be given a unique name: Apache_V11, Apache_V20, etc.
This name must be referred to in the License field.
This allows us to count licenses and file names and build a 3rd-party README
as well as NBMs.
Make sure that the license for a new binary is correctly included under nbbuild/licenses.
If there is no existing license of the same type, it must be reviewed prior to committing.
NetBeans Samples
If a sample is created for NetBeans itself,
it can be packaged into ZIP file and should not be in the external/ folder.
To ensure tests correctly skip over it,
the owner must add an entry for the binary into nbbuild/antsrc/org/netbeans/nbbuild/extlibs/ignored-binaries
and include a brief explanatory comment.
Alternately, it may be preferable to keep the sample files unpacked directly in Hg,
and create the ZIP during the module’s build process
(either directly into the cluster, or into build/classes for inclusion inside the module).
This not only prevents tests from warning about it,
but can make it easier to update minor parts of a sample
and may make version control operations more pleasant.
The sample itself must be covered by the BSD license; the license must be included in every file (excepting binaries such as icons).
Copyright (c) <YEAR>, Sun Microsystems, Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or
without modification, are permitted provided that the following
conditions are met:
* Redistributions of source code must retain the above
copyright notice, this list of conditions and the following
disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials
provided with the distribution.
* Neither the name of Sun Microsystems, Inc. nor the names of
its contributors may be used to endorse or promote products
derived from this software without specific prior written
permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.If sample is not created solely for NetBeans, e.g. bundled in a third-party product and covered by a separate license, it must follow the same rules as for any other binary library.
-
Common mistakes
A binary file has no associated license. (E.g. xyz.jar is missing xyz-license.txt.)
A binary file has an associated license, but does not maintain the naming convention,
or has typos. (E.g. xyz.jar with xy-license.txt.)
Licenses are not pure text. (E.g. they contain HTML.)
A binary file is duplicated in several places. Before adding a new library, please make sure that library is not already available in the Hg tree. If it is, check if the version there is suitable for you; if so, communicate with the owner regarding possible upgrades and/or available packages if they are not available. You might need to move the library to a parent cluster as well. If you do depend on such a third cluster, make sure your module is marked as eager, otherwise it will get disabled.
The names of the binary and its license file will change when the binary is
upgraded to a newer version. Update project.properties
(or, less commonly, build.xml) to reflect this change.
Before moving from my own repository to NetBeans Hg, I used release/modules/ext/ for storing my binary libraries. They need to be moved into external/ unless the library itself is covered by CDDL, build script, licenses etc., must be updated accordingly!
How do I know if some other modules is relying on the source location of my external binaries?
Answer: it’s not hard to find out. For example, if you want to know who uses
httpserver/external, try this (Unix / Bash syntax):
cd nb-main
for f in */{build.xml,nbproject/*.{properties,xml`; \
do fgrep -H httpserver/external $f; doneImplementation work
Interesting files from build:
Static verification of Hg
Part of regular build. Only pays attention to Hg-controlled files in the checkout, so can run on a built source tree without becoming confused. Writes results in JUnit format for easy browsing from Hudson.
-
Look for
.jarnot in/external/dirs (with some exceptions). -
Every license file has at least mandatory headers.
-
Every license file has lines at most 80 characters long, etc.
-
For LGPL, must have
Sourceheader. -
Check that every external
.jaror.ziphas a matching license. (Or it can be mentioned inFilesheader of some license.) -
Every binary has a version number in name.
-
No binary occurs more than once, under any name (so check by CRC-32 or SHA-1 etc.). (Look inside ZIP files for nested JARs.)
-
Every license file’s
Licensefield refers to something innbbuild/licenses. -
The file in
nbbuild/licensesexactly matches the body of the license file. Whitespace-only changes are permitted, e.g. rewrapping lines to make them fit. For licenses with templates (e.g. BSD License) any tokens between two underscores can match whatever character sequence.
Things done in IDE build
Generate a third-party JAR & license summary.
Find every binary in the IDE build which is either
present directly in some /external dir
or present inside a ZIP in some /external dir.
For every such binary, retrieve the license from nbbuild/licenses.
Make a single document listing all of the binaries and licenses.
Verify that no such binary is present in more than one place.
Saved as THIRDPARTYLICENSE-generated.txt in development builds.
