aboutsummaryrefslogtreecommitdiff
path: root/utils/scanpypi
Commit message (Collapse)AuthorAgeFilesLines
* utils/scanpypi: use python3 explicitlyGravatar Thomas Petazzoni2021-02-201-1/+1
| | | | | | | | | | | | | scanpypi is python3 compatible. In addition, it executes the setup.py of Python modules to extract the relevant information. Since these are more and more commonly using python3 constructs, using "python" to run scanpypi causes problems on systems that have python2 installed as python, when trying to parse setup.py scripts with python3 constructs. Fixes part of #13516. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
* utils/scanpypi: use raw strings in re.compile/re.subGravatar Thomas Petazzoni2020-08-141-4/+4
| | | | | | | | | | Fixes the following Python 3.x flake8 warning: W605 invalid escape sequence '\w' Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* utils/scanpypi: update hash file indentation formattingGravatar James Hilliard2020-02-261-3/+3
| | | | | | | The new .hash convention is to use 2 spaces between fields. Signed-off-by: James Hilliard <james.hilliard1@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* utils/scanpypi: remind developer about updating DEVELOPERS and Config.inGravatar Matt Weber2019-12-261-0/+2
| | | | | Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* utils/scanpypi: sort alphabetically list of required packagesGravatar Bartosz Bilas2019-11-081-0/+1
| | | | | | | | | | | | | | That change will alphabetically set list order of required packages in Config.in file automatically. Example below: before: ['python-pyserial', 'python-pyaes', 'python-ecdsa'] after: ['python-ecdsa', 'python-pyaes', 'python-pyserial'] Signed-off-by: Bartosz Bilas <b.bilas@grinn-global.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* scanpypi: write every license file onceGravatar Asaf Kahlon2019-10-111-0/+2
| | | | | | | | | | | | On some cases, when the package contains multiple license files and some of them from the same type, the scanpypi script will write the same license type more than once under _LICENSE. Hence, before creating the _LICENSE variable, we'll remove every possible duplication. Signed-off-by: Asaf Kahlon <asafka7@gmail.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* utils/scanpypi: handle underscores in python packagesGravatar James Hilliard2019-03-011-0/+7
| | | | | | | | | Some python packages seem to use underscores in inconsistent ways. We can attempt to normalize these by always using dashes for the buildroot name and attempting to autodetect the correct metadata name format. Signed-off-by: James Hilliard <james.hilliard1@gmail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* utils/scanpypi: protect against zip-slip vulnerability in zip/tar handlingGravatar Peter Korsgaard2019-02-121-0/+18
| | | | | | | | | | | | | | | | For details, see https://github.com/snyk/zip-slip-vulnerability Older python versions do not validate that the extracted files are inside the target directory. Detect and error out on evil paths before extracting .zip / .tar file. Given the scope of this (zip issue was fixed in python 2.7.4, released 2013-04-06, scanpypi is only used by a developer when adding a new python package), the security impact is fairly minimal, but it is good to get it fixed anyway. Reported-by: Bas van Schaik <security-reports@semmle.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* utils/scanpypi: write _SOURCE only when neededGravatar Asaf Kahlon2018-12-301-7/+8
| | | | | | | | | | | For some packages, there's no need to add the _SOURCE variable, since the name of the source file is the same as the name of the package (like python-engineio). Hence, we'll add it to the .mk file only if needed. Signed-off-by: Asaf Kahlon <asafka7@gmail.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* utils/scanpypi: import third party modules after the standard onesGravatar Yegor Yefremov2018-12-101-3/+3
| | | | | | | | Move imports from six package after the standard modules. Resolves pylint warnings. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* utils/scanpypi: use archive file name to specify the extraction folderGravatar Yegor Yefremov2018-11-021-9/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | Some packages have archive name that is different from package name. For example websocket-client's archive name is websocket_client-*.tar.gz. scanpypi expects the temporary extract folder to be: /tmp-folder/BR-package-name/PyPI-packagename-and-version In the case of websocket-client package the real extraction folder will be different from the expected one because of the '_' in the archive file name. Use archive file name instead of package name to specify the extraction folder. As the version is already part of this file, we don't need to specify it. Bonus: remove obsolete "return None, None" as the function doesn't return anything. OSError class doesn't provide "message" member, so replace it with "strerror". Fixes: https://bugs.busybox.net/show_bug.cgi?id=11251 Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Reviewed-by: Asaf Kahlon <asafka7@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* scanpypi: improve BSD licence handlingGravatar Yegor Yefremov2018-10-111-2/+2
| | | | | | | | | | | | When used without spdx_lookup the BSD licence cannot be detected correctly because many Python packages just specify BSD without the exact version in their metadata. So add a special message warning the user instead of the licence id. Bonus: fix typo. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* scanpypi: place a warning into *.mk file if licence id couldn't be detectedGravatar Yegor Yefremov2018-08-311-0/+2
| | | | | | | | | | If a license file could be found, but license id couldn't be detected place following warning into *.mk file: FOO_LICENSE = FIXME: license id couldn't be detected Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* scanpypi: rework download_package error handlingGravatar Yegor Yefremov2018-06-151-5/+9
| | | | | | | | | | | | Some packages don't provide source archive but only a wheel file. In this case download variable is not defined. So define this variable at the very beginning and check whether it is None after searching for source archives in the metadata. Bonus: fix PEP8 issue with wrong indentation. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* scanpypi: add support for the new PyPI infrastructureGravatar Yegor Yefremov2018-04-181-14/+15
| | | | | | | | | | | https://pypi.python.org URL has been changed to https://pypi.org. Package's JSON object now contains sha256 checksum, so use it instead of locally computed one. Change comments in the hash file accordingly. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* scanpypi: generate help text compliant to check-packageGravatar Ricardo Martincoski2018-04-011-1/+1
| | | | | | | | | Each line must fit in <tab><2 spaces><62 chars>. The default width for textwrap.wrap() is 70, so explicit set it to 62. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* utils/scanpypi: don't hardcode python2Gravatar Peter Korsgaard2018-04-011-1/+1
| | | | | | | | | | | | Commit 3a0c20c5309b (scanpypi: add support for Python3) adapted the script to work with python 3.x, but the shebang still said python2 making it unlikely to work on systems without python 2.x. Change it to just 'python' instead. Signed-off-by: Peter Korsgaard <peter@korsgaard.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* scanpypi: fix code styleGravatar Ricardo Martincoski2018-03-131-1/+3
| | | | | | | | | | Fix these warnings: E401 multiple imports on one line Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Yegor Yefremov <yegorslists@googlemail.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* utils/scanpypi: correctly handle license dirs in subdirs for .hash filesGravatar Peter Korsgaard2018-03-091-1/+1
| | | | | | | | | | | | | create_hash_file() used basename(licensefile) when it writes the entry for the license file in the .hash, which is obviously not correct when license file is locate in a sub directory. Instead copy the logic from __create_mk_license() to strip the directory prefix from the absolute filename instead. Signed-off-by: Peter Korsgaard <peter@korsgaard.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* Merge branch 'next'Gravatar Peter Korsgaard2018-03-051-19/+28
|\ | | | | | | Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
| * scanpypi: fix licence detection handling for unknown licencesGravatar Yegor Yefremov2018-02-261-1/+1
| | | | | | | | | | | | | | Check for match object not being None. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
| * scanpypi: fix Py2/3 conversion leftoverGravatar Yegor Yefremov2018-02-261-1/+1
| | | | | | | | | | | | | | Use urlparse from six package. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
| * scanpypi: add support for Python3Gravatar Yegor Yefremov2018-02-251-17/+26
| | | | | | | | | | | | | | | | The script was changed via modernize utility. The only manual made part was the handling of StringIO. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* | utils/scanpypi: fix 'downloas' typo in error messageGravatar Peter Korsgaard2018-03-011-1/+1
|/ | | | Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* scanpypi: fix code styleGravatar Ricardo Martincoski2018-01-291-4/+9
| | | | | | | | | | | | | | | | | | | | Fix these warnings: E101 indentation contains mixed spaces and tabs E128 continuation line under-indented for visual indent E231 missing whitespace after ',' E261 at least two spaces before inline comment E302 expected 2 blank lines, found 1 E305 expected 2 blank lines after class or function definition, found 1 W191 indentation contains tabs Ignore these warnings: E402 module level import not at top of file Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Yegor Yefremov <yegorslists@googlemail.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* scanpypi: ignore empty elements in package requirementsGravatar Yegor Yefremov2018-01-261-1/+2
| | | | | | | | | | Depending on how setup.py reads requirements files empty elements can occur. This patch takes care, that such elements will be ignored and don't crash the scanpypi script. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Tested-by: Matt Weber <matthew.weber@rockwellcollins.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* scanpypi: get rid of commented lines and also strip the package stringsGravatar Yegor Yefremov2018-01-181-0/+4
| | | | | Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* scanpypi: add support for licence files checksumsGravatar Yegor Yefremov2018-01-151-4/+21
| | | | | | | | | | Store the list of detected licence files in the main object and automatically add their sha256 checksums when creating *.hash file. Bonus: fix wrong indentation. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* scanpypi: get license names from SPDX databaseGravatar Yegor Yefremov2018-01-121-55/+79
| | | | | | | | | | | | Use spdx_lookup package to compare packages' license file texts with SPDX database. This feature is optional. Bonus: fix wrong indentation. Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* utils/scanpypi: increase error message verbosityGravatar Alexey Roslyakov2017-12-181-3/+3
| | | | | | | | When package installation fails it is good to know what happened. Signed-off-by: Alexey Roslyakov <alexey.roslyakov@gmail.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* utils/scanpypi: include LICENSE.RST to supported license filesGravatar Alexey Roslyakov2017-07-201-2/+2
| | | | | | Signed-off-by: Alexey Roslyakov <alexey.roslyakov@gmail.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* utils/scanpypi: don't pass any arguments to main()Gravatar Alexey Roslyakov2017-07-201-2/+2
| | | | | | | | | | | | 'if __name__ == "__main__"' idiom typically calls main function that doesn't take any arguments in most cases. We shouldn't pass any tuple to it. I've tested the script with python-idna-2.5 and now it works with this little change. Signed-off-by: Alexey Roslyakov <alexey.roslyakov@gmail.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* tools: rename to 'utils'Gravatar Thomas Petazzoni2017-07-011-0/+653
After some discussion, we found out that "tools" has the four first letters identical to the "toolchain" subfolder, which makes it a bit unpractical with tab-completion. So, this commit renames "tools" to "utils", which is more tab-completion-friendly. This has been discussed with Arnout and Yann. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>