rebillar/site-accueil-insa
1
1
Fork 1
forked from vergnet/site-accueil-insa
Code Issues Pull requests Releases Wiki Activity

total photo count for zip files

Browse source
This commit is contained in:
Baptiste Rebillard 2022-10-26 08:59:36 +02:00
parent 0bede532c2
commit 86124f4f40
9719 changed files with 1024941 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
How to install Matomo.html Normal file
View file

@ -0,0 +1,8 @@
<html>
<head>
<meta http-equiv="refresh" content="0;url=https://matomo.org/docs/installation/"/>
<meta name="robots" content="noindex,nofollow">
</head>
<body>You will be redirected to the Matomo Analytics Installation documentation on <a href='https://matomo.org/docs/installation/'>matomo.org/docs/installation</a>
</body>
</html>

1188
matomo/CHANGELOG.md Normal file
View file

File diff suppressed because it is too large Load diff

16
matomo/CONTRIBUTING.md Normal file
View file

@ -0,0 +1,16 @@
# How to contribute
Great to have you here! Read the following guide on our developer zone to learn how you can help make this project better!
https://developer.matomo.org/guides/contributing-to-piwik-core
## How to submit a bug report or suggest a feature?
Please read the recommendations on writing a good [bug report](https://developer.matomo.org/guides/core-team-workflow#submitting-a-bug-report) or [feature request](https://developer.matomo.org/guides/core-team-workflow#submitting-a-feature-request).
## How to suggest improvements to translations?
You can help improve translations in Matomo, please read [contribute to translations](https://github.com/matomo-org/matomo/blob/master/lang/README.md).
## How to submit code improvements via pull requests?
You can help contribute to Matomo codebase via Pull Requests, see [Contributing to Matomo core](https://developer.matomo.org/guides/contributing-to-piwik-core)

27
matomo/DIObject.php Normal file
View file

@ -0,0 +1,27 @@
<?php
namespace DI;
/**
* This file aims to circumvent problems when updating to Matomo 4.
* Matomo 4 includes a newer version of PHP-DI, which does not include \DI\object() any longer
* To not run into any problems with plugins still using that we forward this method to \DI\autowire
*/
if (!function_exists("\DI\object")) {
function object()
{
return call_user_func_array("\DI\autowire", func_get_args());
}
}
if (!function_exists("\DI\link")) {
function link()
{
return call_user_func_array("\DI\get", func_get_args());
}
}

299
matomo/LEGALNOTICE Normal file
View file

@ -0,0 +1,299 @@
COPYRIGHT
Matomo - free/libre analytics platform
The software package is:
Copyright (C) 2014 Matthieu Aubry
Individual contributions, components, and libraries are copyright
of their respective authors.
SOFTWARE LICENSE
The free software license of Matomo is GNU General Public License v3
or later. A copy of GNU GPL v3 should have been included in this
software package in LICENSE.
TRADEMARK
Matomo (TM) is an internationally registered trademark.
The software license does not grant any rights under trademark
law for use of the trademark. Refer to https://matomo.org/trademark/
for up-to-date trademark licensing information.
*
* The software license applies to both the aggregate software and
* application-specific portions of the software.
*
* You may not remove this legal notice or modify the software
* in such a way that misrepresents the origin of the software.
*
CREDITS
The software consists of contributions made by many individuals.
Major contributors are listed in https://matomo.org/team/.
For detailed contribution history, refer to the source, tickets,
patches, and Git revision history, available at
https://github.com/matomo-org/matomo/issues
https://github.com/matomo-org/matomo
SEPARATELY LICENSED COMPONENTS AND LIBRARIES
The following components/libraries are distributed in this package,
and subject to their respective licenses.
Name: javascriptCode.tpl - tracking tag to embed in your web pages
Link: https://github.com/matomo-org/matomo/blob/master/core/Tracker/javascriptTag.tpl
License: Public Domain
Name: jquery.truncate
Link: https://github.com/matomo-org/matomo/blob/master/libs/jquery/truncate/
License: New BSD
Name: matomo.js & piwik.js - JavaScript tracker
Link: https://github.com/matomo-org/matomo/blob/master/js/piwik.js
Link: https://github.com/matomo-org/matomo/blob/master/js/matomo.js
License: New BSD
Name: PiwikTracker - server-side tracker (PHP)
Link: https://github.com/matomo-org/matomo/blob/master/libs/PiwikTracker/
License: New BSD
Name: DeviceDetector
Link: https://github.com/matomo-org/device-detector
License: LGPL
Name: Piwik/Decompress
Link: https://github.com/matomo-org/component-decompress
License: LGPL v3.0
Name: Piwik/Network
Link: https://github.com/matomo-org/component-network
License: LGPL v3.0
THIRD-PARTY COMPONENTS AND LIBRARIES
The following components/libraries are redistributed in this package,
and subject to their respective licenses.
Name: jqPlot
Link: http://www.jqplot.com/
License: Dual-licensed: MIT (Expat) or GPL v2
Name: jQuery
Link: https://jquery.com/
License: Dual-licensed: MIT (Expat) or GPL
Notes:
- GPL version not explicitly stated in source but GPL v2 is in git
- includes Sizzle.js - multi-licensed: MIT (Expat), New BSD, or GPL [v2]
Name: jQuery UI
Link: https://jqueryui.com/
License: Dual-licensed: MIT (Expat) or GPL
Notes:
- GPL version not explicitly stated in source but GPL v2 is in git
Name: jquery.history
Link: https://tkyk.github.io/jquery-history-plugin/
License: MIT (Expat)
Name: jquery.scrollTo
Link: http://plugins.jquery.com/project/ScrollTo
License: Dual licensed: MIT (Expat) or GPL
Name: jquery Tooltip
Link: http://bassistance.de/jquery-plugins/jquery-plugin-tooltip/
License: Dual licensed: MIT (Expat) or GPL
Name: jquery placeholder
Link: http://mths.be/placeholder
License: Dual licensed: MIT (Expat) or GPL
Name: qrcode.js
Link: https://github.com/davidshimjs/qrcodejs
License: MIT
Name: json2.js
Link: http://json.org/
License: Public domain
Notes:
- reference implementation
Name: jshrink
Link: https://github.com/tedivm/jshrink
License: BSD-3-Clause
Name: Sparkline
Link: https://github.com/jamiebicknell/Sparkline
License: MIT
Name: sprintf
Link: http://www.diveintojavascript.com/projects/javascript-sprintf
License: New BSD
Name: upgrade.php
Link: http://upgradephp.berlios.de/
License: Public domain
Name: Archive Tar
Link: https://pear.php.net/package/Archive_Tar
License: New BSD
Name: Event Dispatcher (and Notification)
Link: https://pear.php.net/package/Event_Dispatcher/
License: New BSD
Name: HTML Common2
Link: https://pear.php.net/package/HTML_Common2/
License: New BSD
Name: HTML QuickForm2
Link: https://pear.php.net/package/HTML_QuickForm2/
License: New BSD
Name: HTML QuickForm2_Renderer_Smarty
Link: http://www.phcomp.co.uk/tmp/Smarty.phps
License: New BSD
Name: MaxMindGeoIP
Link: https://dev.maxmind.com/geoip/legacy/downloadable/#PHP-7
License: LGPL
Name: PclZip
Link: http://www.phpconcept.net/pclzip/
License: LGPL
Notes:
- GPL version not explicitly stated but tarball contains LGPL v2.1
Name: PEAR (base system)
Link: https://pear.php.net/package/PEAR
License: New BSD
Name: PhpSecInfo
Link: http://phpsec.org/projects/phpsecinfo/
License: New BSD
Name: RankChecker
Link: http://www.getrank.org/free-pagerank-script
License: GPL
Name: Twig
Link: https://twig.symfony.com/
License: BSD
Name: TCPDF
Link: https://sourceforge.net/projects/tcpdf/
License: LGPL v3 or later
Name: Zend Framework
Link: https://framework.zend.com/
License: New BSD
Name: pChart 2.1.4
Link: http://www.pchart.net
License: GPL v3
Name: Chroma.js
Link: https://github.com/gka/chroma.js
License: GPL v3
Name: qTip2 - Pretty powerful tooltips
Link: http://craigsworks.com/projects/qtip2/
License: GPL
Name: Kartograph.js
Link: http://kartograph.org/
License: LGPL v3 or later
Name: Raphaël - JavaScript Vector Library
Link: http://raphaeljs.com/
License: MIT (Expat)
Name: iFrame Resizer
Link: https://github.com/davidjbradshaw/iframe-resizer
License: MIT
Name: lessphp
Link: http://leafo.net/lessphp
License: GPL3, MIT (Expat)
Name: Symfony Console Component
Link: https://github.com/symfony/Console
License: MIT (Expat)
Name: AngularJS
Link: https://github.com/angular/angular.js
License: MIT (Expat)
Name: Mousetrap
Link: https://github.com/ccampbell/mousetrap
License: Apache 2.0
Name: PHP-DI
Link: http://php-di.org/
License: MIT (Expat)
THIRD-PARTY CONTENT
Name: FamFamFam icons - Mark James
Link: http://www.famfamfam.com/lab/icons/
License: CC BY 3.0
Name: Solar System icons - Dan Wiersema
Link: http://www.iconspedia.com/icon/neptune-4672.html
License: Free for non-commercial use
Notes:
- used in Matomo's ExampleUI plugin
Name: flag-icon-css - Lipis
Link: https://github.com/lipis/flag-icon-css
License: MIT (Expat)
Notes:
- used for flag PNGs
Name: Wine project - tahoma.ttf font
Link: http://source.winehq.org/git/wine.git/blob_plain/HEAD:/fonts/tahoma.ttf
License: LGPL v2.1
Notes:
- used in ImageGraph plugin
Name: plugins/Feedback/angularjs/ratefeature/thumbs-down.png
Link: https://www.iconfinder.com/icons/216428/down_thumbs_icon
License: Creative Commons (Attribution-Share Alike 3.0 Unported)
Name: plugins/Feedback/angularjs/ratefeature/thumbs-up.png
Link: https://www.iconfinder.com/icons/216429/thumbs_up_icon
License: Creative Commons (Attribution-Share Alike 3.0 Unported)
Name: plugins/Insights/images/idea.png
Link: https://www.iconfinder.com/icons/6074/brainstorm_bulb_idea_jabber_light_icon
License: GPL
By: Alessandro Rei - http://www.kde-look.org/usermanager/search.php?username=mentalrey
Name: Material icons ("icon-info2", "icon-outline", "icon-settings", "icon-form", "icon-play", "icon-pause", "icon-replay", "icon-skip-next", "icon-skip-forward", "icon-stop", "icon-fast-forward", "icon-fast-rewind", "icon-bug", "icon-upload", "icon-segmented-visits-log", "icon-clock") in plugins/Morpheus/fonts, and plugins/Morpheus/images/compare.svg
Link: https://design.google.com/icons/
License: Apache License Version 2.0
Name: IcoMoon - Free icons ("icon-funnel", "icon-lab", "icon-archive", "icon-rocket", "icon-embed", "icon-page-performance", "icon-github") in plugins/Morpheus/fonts
Link: https://icomoon.io/#icons-icomoon
License: GPL
Name: FontAwesome icons ("icon-minus-square", "icon-plus-square", "icon-arrowup", "icon-arrowdown", "icon-outlink") in plugins/Morpheus/fonts
Link: https://fontawesome.com/
License: CC BY 4.0
Notes:
- the "New BSD" license refers to either the "Modified BSD" and "Simplified BSD"
licenses (2- or 3-clause), which are GPL compatible.
- icons for browsers, operating systems, browser plugins, brands, search engines, social media websites
and flags of countries are nominative use of third-party trademarks when
referring to the corresponding product or entity

675
matomo/LICENSE Normal file
View file

@ -0,0 +1,675 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
{one line to give the program's name and a brief idea of what it does.}
Copyright (C) {year} {name of author}
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
Matomo Copyright (C) 2007-2018 Matomo.org
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<http://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<http://www.gnu.org/philosophy/why-not-lgpl.html>.

31
matomo/LegacyAutoloader.php Normal file
View file

@ -0,0 +1,31 @@
<?php
class LegacyAutoloader
{
public function __construct()
{
spl_autoload_register(array($this, 'load_class'));
}
public static function register()
{
new LegacyAutoloader();
}
public function load_class($className)
{
if (strpos($className, 'Matomo\\') === 0) {
$newName = 'Piwik' . substr($className, 6);
if (class_exists($newName) && !class_exists($className, false)) {
@class_alias($newName, $className);
}
} elseif (strpos($className, 'Piwik\\') === 0) {
$newName = 'Matomo' . substr($className, 5);
if (class_exists($newName) && !class_exists($className, false)) {
@class_alias($newName, $className);
}
}
}
}
LegacyAutoloader::register();

60
matomo/PRIVACY.md Normal file
View file

@ -0,0 +1,60 @@
# Privacy
This is a summary of all of the components within Matomo which may affect your privacy in some way. Please keep in mind
third party Themes, Plugins or Apps may introduce privacy concerns not listed here.
## Privacy for users being tracked by Matomo
In this section we document how to protect the privacy of visitors who are tracked by your Matomo analytics service.
### Anonymise visitor IP addresses
By default, Matomo stores the visitor IP address (IPv4 or IPv6 format) in the database for each new visitor.
If a visitor has a static IP address this means their browsing history can be easily identified across several days and
even across several websites tracked within the same Matomo server. You can anonymize IP addresses to ensure visitors cannot
be tracked this way: [How to anonymise IP addresses.](https://matomo.org/docs/privacy/#step-1-automatically-anonymize-visitor-ips)
### Delete old visitors logs
By default, Matomo stores tracked data forever. To better respect the privacy of your users, it is recommended to regularly
purge old data. You can configure Matomo to automatically delete log data older than a specified number of months:
[How to delete old visitors log data.](https://matomo.org/docs/privacy/#step-2-delete-old-visitors-logs)
### Include a tracking Opt-Out feature on your site
In your website, we recommended providing an easy way for your visitors to “opt-out” of being tracked by Matomo.
You can use the Opt-Out feature to display a link your website that sets a special browser cookie (`matomo_ignore`) when
clicked. Visitors that click that link will be ignored by Matomo in the future:
[How to include a tracking opt-out iframe.](https://matomo.org/docs/privacy/#step-3-include-a-web-analytics-opt-out-feature-on-your-site-using-an-iframe)
### Respect DoNotTrack preference
Do Not Track is a browser-level technology and policy proposal that lets visitors opt out of tracking by websites they
do not visit. Visitors can enable this preference in their browser, and then it's up to Matomo to respect it. By default,
Matomo is configured to ignore visitors that have enabled it:
[How to check if your Matomo respects DoNotTrack.] (https://matomo.org/docs/privacy/#step-4-respect-donottrack-preference)
### Disable tracking cookies
A cookie is a collection of information that a website stores on a visitor’s computer and accesses each time the visitor
returns. By default, Matomo uses cookies to aid in tracking visitor behavior. If someone gains access to a visitor's
computer, they could learn a few things about how the visitor visited your website. For many websites, this isn't a
problem, but for others where a strong level of privacy is required (like online banking), disabling tracking cookies may
be a good idea: [How to disable tracking cookies.](https://matomo.org/faq/general/faq_157/)
### Keep your visitors details private
Any user that has at least `view` access (the default access level) to Matomo can view detailed information for all users
tracked in Matomo (such as their IP addresses, visitor IDs, details of all past visits and actions, etc.) through features
provided by the `Live` plugin (such as the Visitor Log and Visitor Profile). As the Matomo administrator, you may decide
that not all of your users need access to this data. You can deactivate the `Live` plugin to prevent users from viewing
visitor details in the Administration > Plugins page.
## Privacy for Matomo admins and website owners
In this section we document how a Matomo administrator can better protect their own privacy.
### Keep your Matomo server URL private
By default, the Matomo Javascript code on all tracked websites contains the Matomo server URL. In some cases you might
want to hide this Matomo URL completely while still tracking all websites in your Matomo instance. To hide your Matomo
server's URL, you can modify the Javascript Tracking code and point it to a proxy piwik.php script instead of your actual
Matomo server: [How to keep Matomo server URL private.](https://matomo.org/faq/how-to/faq_132/)
### Automatic update check
From time to time, Matomo uses `api.matomo.org` to check if the current version of Matomo is the latest version of Matomo.
If an update is available, a notification is displayed allowing you to upgrade Matomo. To disable the update check,
and stop your instance from sending HTTP requests to `api.matomo.org`, deactivate the "Automatic update" feature by
setting `enable_auto_update = 0` in your configuration file `config/config.ini.php`.
Learn more about [Privacy in Matomo](https://matomo.org/privacy/).

125
matomo/README.md Normal file
View file

@ -0,0 +1,125 @@
# Matomo (formerly Piwik) - matomo.org
[![Latest Stable Version](https://poser.pugx.org/matomo/matomo/v/stable)](https://matomo.org/download/)
[![Latest Unstable Version](https://poser.pugx.org/matomo/matomo/v/unstable)](https://builds.matomo.org/)
[![License](https://poser.pugx.org/piwik/piwik/license)](https://matomo.org/free-software/)
## Code Status
[![Build Status](https://travis-ci.com/matomo-org/matomo.svg?branch=4.x-dev)](https://app.travis-ci.com/matomo-org/matomo/branches)
[![Percentage of issues still open](http://isitmaintained.com/badge/open/matomo-org/matomo.svg)](http://isitmaintained.com/project/matomo-org/matomo "Percentage of issues still open")
## Description
Matomo is the leading Free/Libre open analytics platform.
Matomo is a full-featured PHP MySQL software program that you download and install on your own webserver.
At the end of the five-minute installation process, you will be given a JavaScript code.
Simply copy and paste this tag on websites you wish to track and access your analytics reports in real-time.
Matomo aims to be a Free software alternative to Google Analytics and is already used on more than 1,400,000 websites. Privacy is built-in!
## Mission Statement
> « To create, as a community, the leading international open source digital analytics platform, that gives every user full control of their data. »
Or in short:
> « Liberate Web Analytics »
## License
Matomo is released under the GPL v3 (or later) license, see [LICENSE](LICENSE).
## Requirements
* PHP 7.2.5 or greater
* MySQL version 5.5 or greater, or MariaDB
* PHP extension pdo and pdo_mysql, or the MySQLi extension
* Matomo is OS / server independent
See https://matomo.org/docs/requirements/.
## Install Matomo
* [Download Matomo](https://matomo.org/download/)
* Upload matomo to your webserver
* Point your browser to the directory
* Follow the steps
* Add the given javascript code to your pages
* (You may also generate fake data to experiment, by enabling the plugin VisitorGenerator)
See https://matomo.org/docs/installation/.
(When using Matomo for development you need to [install Matomo from the Git repository](https://matomo.org/faq/how-to-install/faq_18271/)).
## Free trial
If you do not have a server or don't want to host yourself you can use our Matomo Cloud partner service (21 day free trial): https://matomo.org/start-free-analytics-trial/
## Online Demo
Check out the online demo for Matomo at [demo.matomo.cloud](https://demo.matomo.cloud/).
## Changelog
For the list of all tickets closed in the current and past releases, see [matomo.org/changelog/](https://matomo.org/changelog/). For the list of technical changes in the Matomo platform, see [developer.matomo.org/changelog](https://developer.matomo.org/changelog).
## Get involved!
We believe in liberating Web Analytics, providing a free platform for simple and advanced analytics. Matomo was built by dozens of people like you,
and we need your help to make Matomo better… Why not participate in a useful project today? [Learn how you can contribute to Matomo](https://matomo.org/get-involved).
## Translations
Our translations are managed on [Weblate](https://hosted.weblate.org/engage/matomo/).
[![Translation Status](https://hosted.weblate.org/widgets/matomo/-/horizontal-auto.svg)](https://hosted.weblate.org/engage/matomo/)
## Quality Assurance
The Matomo project uses an ever-expanding comprehensive set of thousands of unit tests and hundreds of automated integration tests, system tests, JavaScript tests, and screenshot UI tests, running on a continuous integration server as part of its software quality assurance. [Learn more](https://developer.matomo.org/guides/tests).
We use [BrowserStack.com](https://www.browserstack.com/) testing tool to help check the Matomo user interface is compatible with many browsers.
## Security
Security is a top priority at Matomo. As potential issues are discovered, we validate, patch and release fixes as quickly as we can. We have a security bug bounty program in place that rewards researchers for finding security issues and disclosing them to us.
[Learn more](https://matomo.org/security/) or check out our [HackerOne program](https://hackerone.com/matomo).
## Support for Matomo
For **Free support**, post a message in our community forums: [forum.matomo.org](https://forum.matomo.org/)
For **Professional paid support**, purchase a Matomo On-Premises Support Plan: [matomo.org/support-plans](https://matomo.org/support-plans/)
## Contact
Website: [matomo.org](https://matomo.org)
About us: [matomo.org/team/](https://matomo.org/team/)
Contact us: [matomo.org/contact/](https://matomo.org/contact/)
## More information
What makes Matomo unique from the competition:
* You own your web analytics data: since Matomo is installed on your server, the data is stored in your own database and you can get all the statistics using the powerful Matomo Analytics API.
* Matomo is a Free Software which can easily be configured to respect your visitors' privacy.
* Modern, easy to use User Interface: you can fully customize your dashboard, drag and drop widgets and more.
* Matomo features are built inside plugins: you can add new features and remove the ones you don’t need.
You can build your own web analytics plugins or hire a consultant to have your custom feature built-in Matomo.
* A vibrant international Open community of more than 200,000 active users (tracking even more websites!)
* Advanced Web Analytics capabilities such as E-commerce Tracking, Goal tracking, Campaign tracking,
Custom Variables, Email Reports, Custom Segment Editor, Geo Location, Real-time visits and maps, [and a lot more!](https://matomo.org/feature-overview/)
Documentation and more info on https://matomo.org.
We are together creating the best open analytics platform in the world!

31
matomo/SECURITY.md Normal file
View file

@ -0,0 +1,31 @@
# Reporting Security Issues
## Security Bug Bounty Program
The Matomo Security Bug Bounty Program is designed to encourage security research in Matomo software and to reward those who help us create the safest web analytics platform.
Critical security issues will be rewarded up to 5,000 USD. Critical issue in Matomo means an issue in our latest official release at: https://builds.matomo.org/latest.zip as installed on a typical server (and possibly using any of our official plugins by Matomo or InnoCraft from the Marketplace). If you can gain remote code execution on the server (i.e. RCE), or if you're able to delete data with an HTTPS request (i.e. SQL Injection), this may qualify as a Critical issue.
(Note: If a Remote Code Execution (RCE) is only available when logged in as a Super User, the issue will qualify as "High" and not "Critical".)
High security issues that can cause a direct attack (CSRF, XSS, Auth bypass, etc.) will be rewarded with up to 777 USD.
Other issues will typically be rewarded by 242 USD (or more or less depending on the impact.)
## Responsible disclosure by email
We encourage you to responsibly report issues via our [Matomo Bug Bounty Program on HackerOne](https://hackerone.com/matomo) or you can also
[email us at security@matomo.org](mailto:security@matomo.org?subject=Reporting%20Vulnerability%20in%20Matomo).
If you have found a security issue in Matomo please read [our security notes](https://matomo.org/security/) regarding responsible disclosures.
## Improve your Matomo Server Security
[Secure Matomo server](https://matomo.org/docs/security/): follow these steps to keep your Matomo data safe.
## Security announcements
Please subscribe to [the Changelog](https://matomo.org/changelog/) ([rss feed](https://matomo.org/changelog/feed/)) to be notified of new releases (including security releases).

79
matomo/config/config.ini.php Normal file
View file

@ -0,0 +1,79 @@
; <?php exit; ?> DO NOT REMOVE THIS LINE
; file automatically generated or modified by Matomo; you can manually override the default values in global.ini.php by redefining them in this file.
[database]
host = "127.0.0.1"
username = "accueil_insa"
password = "In2k18"
dbname = "accueil_insa"
tables_prefix = "matomo_"
charset = "utf8mb4"
[General]
salt = "a3c43b7119d9cd3140d5db39cc1b3d98"
trusted_hosts[] = "etud.insa-toulouse.fr"
[PluginsInstalled]
PluginsInstalled[] = "Diagnostics"
PluginsInstalled[] = "Login"
PluginsInstalled[] = "CoreAdminHome"
PluginsInstalled[] = "UsersManager"
PluginsInstalled[] = "SitesManager"
PluginsInstalled[] = "Installation"
PluginsInstalled[] = "Monolog"
PluginsInstalled[] = "Intl"
PluginsInstalled[] = "CoreVue"
PluginsInstalled[] = "CorePluginsAdmin"
PluginsInstalled[] = "CoreHome"
PluginsInstalled[] = "WebsiteMeasurable"
PluginsInstalled[] = "IntranetMeasurable"
PluginsInstalled[] = "CoreVisualizations"
PluginsInstalled[] = "Proxy"
PluginsInstalled[] = "API"
PluginsInstalled[] = "Widgetize"
PluginsInstalled[] = "Transitions"
PluginsInstalled[] = "LanguagesManager"
PluginsInstalled[] = "Actions"
PluginsInstalled[] = "Dashboard"
PluginsInstalled[] = "MultiSites"
PluginsInstalled[] = "Referrers"
PluginsInstalled[] = "UserLanguage"
PluginsInstalled[] = "DevicesDetection"
PluginsInstalled[] = "Goals"
PluginsInstalled[] = "Ecommerce"
PluginsInstalled[] = "SEO"
PluginsInstalled[] = "Events"
PluginsInstalled[] = "UserCountry"
PluginsInstalled[] = "GeoIp2"
PluginsInstalled[] = "VisitsSummary"
PluginsInstalled[] = "VisitFrequency"
PluginsInstalled[] = "VisitTime"
PluginsInstalled[] = "VisitorInterest"
PluginsInstalled[] = "RssWidget"
PluginsInstalled[] = "Feedback"
PluginsInstalled[] = "TwoFactorAuth"
PluginsInstalled[] = "CoreUpdater"
PluginsInstalled[] = "CoreConsole"
PluginsInstalled[] = "ScheduledReports"
PluginsInstalled[] = "UserCountryMap"
PluginsInstalled[] = "Live"
PluginsInstalled[] = "PrivacyManager"
PluginsInstalled[] = "ImageGraph"
PluginsInstalled[] = "Annotations"
PluginsInstalled[] = "MobileMessaging"
PluginsInstalled[] = "Overlay"
PluginsInstalled[] = "SegmentEditor"
PluginsInstalled[] = "Insights"
PluginsInstalled[] = "Morpheus"
PluginsInstalled[] = "Contents"
PluginsInstalled[] = "BulkTracking"
PluginsInstalled[] = "Resolution"
PluginsInstalled[] = "DevicePlugins"
PluginsInstalled[] = "Heartbeat"
PluginsInstalled[] = "Marketplace"
PluginsInstalled[] = "ProfessionalServices"
PluginsInstalled[] = "UserId"
PluginsInstalled[] = "CustomJsTracker"
PluginsInstalled[] = "Tour"
PluginsInstalled[] = "PagePerformance"
PluginsInstalled[] = "CustomDimensions"

12
matomo/config/environment/dev.php Normal file
View file

@ -0,0 +1,12 @@
<?php
return array(
'Matomo\Cache\Backend' => DI\autowire('Matomo\Cache\Backend\ArrayCache'),
'Piwik\Translation\Loader\LoaderInterface' => DI\autowire('Piwik\Translation\Loader\LoaderCache')
->constructorParameter('loader', DI\get('Piwik\Translation\Loader\DevelopmentLoader')),
'Piwik\Translation\Loader\DevelopmentLoader' => DI\create()
->constructor(DI\get('Piwik\Translation\Loader\JsonFileLoader')),
);

1260
matomo/config/global.ini.php Normal file
View file

File diff suppressed because it is too large Load diff

239
matomo/config/global.php Normal file
View file

@ -0,0 +1,239 @@
<?php
use Psr\Container\ContainerInterface;
use Matomo\Cache\Eager;
use Piwik\SettingsServer;
return [
'path.root' => PIWIK_DOCUMENT_ROOT,
'path.misc.user' => 'misc/user/',
'path.tmp' => function (ContainerInterface $c) {
$root = PIWIK_USER_PATH;
// TODO remove that special case and instead have plugins override 'path.tmp' to add the instance id
if ($c->has('ini.General.instance_id')) {
$instanceId = $c->get('ini.General.instance_id');
$instanceId = $instanceId ? '/' . $instanceId : '';
} else {
$instanceId = '';
}
/** @var Piwik\Config\ $config */
$config = $c->get('Piwik\Config');
$general = $config->General;
$tmp = empty($general['tmp_path']) ? '/tmp' : $general['tmp_path'];
return $root . $tmp . $instanceId;
},
'path.tmp.templates' => DI\string('{path.tmp}/templates_c'),
'path.cache' => DI\string('{path.tmp}/cache/tracker/'),
'view.clearcompiledtemplates.enable' => true,
'twig.cache' => DI\string('{path.tmp.templates}'),
'Matomo\Cache\Eager' => function (ContainerInterface $c) {
$backend = $c->get('Matomo\Cache\Backend');
$cacheId = $c->get('cache.eager.cache_id');
if (SettingsServer::isTrackerApiRequest()) {
$eventToPersist = 'Tracker.end';
$cacheId .= 'tracker';
} else {
$eventToPersist = 'Request.dispatch.end';
$cacheId .= 'ui';
}
$cache = new Eager($backend, $cacheId);
\Piwik\Piwik::addAction($eventToPersist, function () use ($cache) {
$cache->persistCacheIfNeeded(43200);
});
return $cache;
},
'Matomo\Cache\Backend' => function (ContainerInterface $c) {
// If Piwik is not installed yet, it's possible the tmp/ folder is not writable
// we prevent failing with an unclear message eg. coming from doctrine-cache
// by forcing to use a cache backend which always works ie. array
if (!\Piwik\SettingsPiwik::isMatomoInstalled()) {
$backend = 'array';
} else {
try {
$backend = $c->get('ini.Cache.backend');
} catch (\DI\NotFoundException $ex) {
$backend = 'chained'; // happens if global.ini.php is not available
}
}
return \Piwik\Cache::buildBackend($backend);
},
'cache.eager.cache_id' => function () {
return 'eagercache-' . str_replace(['.', '-'], '', \Piwik\Version::VERSION) . '-';
},
'entities.idNames' => DI\add(['idGoal', 'idDimension']),
'Psr\Log\LoggerInterface' => DI\create('Psr\Log\NullLogger'),
'Piwik\Translation\Loader\LoaderInterface' => DI\autowire('Piwik\Translation\Loader\LoaderCache')
->constructorParameter('loader', DI\get('Piwik\Translation\Loader\JsonFileLoader')),
'DeviceDetector\Cache\Cache' => DI\autowire('Piwik\DeviceDetector\DeviceDetectorCache')->constructor(86400),
'observers.global' => [],
/**
* By setting this option to false, the check that the DB schema version matches the version of the source code will
* be no longer performed. Thus it allows you to execute for example a newer version of Matomo with an older Matomo
* database version. Please note disabling this setting is not recommended because often an older DB version is not
* compatible with newer source code.
* If you disable this setting, make sure to execute the updates after updating the source code. The setting can be
* useful if you want to update Matomo without any outage when you know the current source code update will still
* run fine for a short time while in the background the database updates are running.
*/
'EnableDbVersionCheck' => true,
'fileintegrity.ignore' => DI\add([
'*.htaccess',
'*web.config',
'bootstrap.php',
'favicon.ico',
'robots.txt',
'.bowerrc',
'.lfsconfig',
'.phpstorm.meta.php',
'config/config.ini.php',
'config/config.php',
'config/common.ini.php',
'config/*.config.ini.php',
'config/manifest.inc.php',
'misc/*.dat',
'misc/*.dat.gz',
'misc/*.mmdb',
'misc/*.mmdb.gz',
'misc/*.bin',
'misc/user/*png',
'misc/user/*svg',
'misc/user/*js',
'misc/user/*/config.ini.php',
'misc/package',
'misc/package/WebAppGallery/*.xml',
'misc/package/WebAppGallery/install.sql',
'plugins/ImageGraph/fonts/unifont.ttf',
'plugins/*/config/tracker.php',
'plugins/*/config/config.php',
'vendor/autoload.php',
'vendor/composer/autoload_real.php',
'vendor/szymach/c-pchart/app/*',
'tmp/*',
// Search engine sites verification
'google*.html',
'BingSiteAuth.xml',
'yandex*.html',
// common files on shared hosters
'php.ini',
'.user.ini',
'error_log',
// Files below are not expected but they used to be present in older Piwik versions and may be still here
// As they are not going to cause any trouble we won't report them as 'File to delete'
'*.coveralls.yml',
'*.scrutinizer.yml',
'*.gitignore',
'*.gitkeep',
'*.gitmodules',
'*.gitattributes',
'*.bower.json',
'*.travis.yml',
]),
'Piwik\EventDispatcher' => DI\autowire()->constructorParameter('observers', DI\get('observers.global')),
'login.allowlist.ips' => function (ContainerInterface $c) {
/** @var Piwik\Config\ $config */
$config = $c->get('Piwik\Config');
$general = $config->General;
$ips = [];
if (!empty($general['login_allowlist_ip']) && is_array($general['login_allowlist_ip'])) {
$ips = $general['login_allowlist_ip'];
} elseif (!empty($general['login_whitelist_ip']) && is_array($general['login_whitelist_ip'])) {
// for BC
$ips = $general['login_whitelist_ip'];
}
$ipsResolved = [];
foreach ($ips as $ip) {
$ip = trim($ip);
if (filter_var($ip, FILTER_VALIDATE_IP) || \Matomo\Network\IPUtils::getIPRangeBounds($ip) !== null) {
$ipsResolved[] = $ip;
} else {
$lazyCache = \Piwik\Cache::getLazyCache();
$cacheKey = 'DNS.' . md5($ip);
$resolvedIps = $lazyCache->fetch($cacheKey);
if (!is_array($resolvedIps)) {
$resolvedIps = [];
$ipFromHost = @gethostbyname($ip);
if (!empty($ipFromHost) && $ipFromHost !== $ip) {
$resolvedIps[] = $ipFromHost;
}
if (function_exists('dns_get_record')) {
$entry = @dns_get_record($ip, DNS_AAAA);
if (
!empty($entry['0']['ipv6'])
&& filter_var($entry['0']['ipv6'], FILTER_VALIDATE_IP)
) {
$resolvedIps[] = $entry['0']['ipv6'];
}
}
$lazyCache->save($cacheKey, $resolvedIps, 30);
}
$ipsResolved = array_merge($ipsResolved, $resolvedIps);
}
}
return $ipsResolved;
},
/**
* This defines a list of hostnames Matomo's Http class will deny requests to. Wildcards (*) can be used in the
* beginning to match any subdomain level or in the end to match any tlds
*/
'http.blocklist.hosts' => [
'*.amazonaws.com',
],
'Piwik\Tracker\VisitorRecognizer' => DI\autowire()
->constructorParameter('trustCookiesOnly', DI\get('ini.Tracker.trust_visitors_cookies'))
->constructorParameter('visitStandardLength', DI\get('ini.Tracker.visit_standard_length'))
->constructorParameter('lookbackNSecondsCustom', DI\get('ini.Tracker.window_look_back_for_visitor')),
'Piwik\Tracker\Settings' => DI\autowire()
->constructorParameter(
'isSameFingerprintsAcrossWebsites',
DI\get('ini.Tracker.enable_fingerprinting_across_websites')
),
'archiving.performance.logger' => null,
\Piwik\CronArchive\Performance\Logger::class => DI\autowire()
->constructorParameter('logger', DI\get('archiving.performance.logger')),
\Piwik\Concurrency\LockBackend::class => \DI\get(\Piwik\Concurrency\LockBackend\MySqlLockBackend::class),
\Piwik\Segment\SegmentsList::class => function () {
return \Piwik\Segment\SegmentsList::get();
}
];

9712
matomo/config/manifest.inc.php Normal file
View file

File diff suppressed because it is too large Load diff

32
matomo/console Normal file
View file

@ -0,0 +1,32 @@
#!/usr/bin/env php
<?php
use Piwik\FrontController;
if (!defined('PIWIK_DOCUMENT_ROOT')) {
define('PIWIK_DOCUMENT_ROOT', dirname(__FILE__) == '/' ? '' : dirname(__FILE__));
}
if (file_exists(PIWIK_DOCUMENT_ROOT . '/bootstrap.php')) {
require_once PIWIK_DOCUMENT_ROOT . '/bootstrap.php';
}
if (!defined('PIWIK_INCLUDE_PATH')) {
define('PIWIK_INCLUDE_PATH', PIWIK_DOCUMENT_ROOT);
}
require_once PIWIK_INCLUDE_PATH . '/core/bootstrap.php';
if (!Piwik\Common::isPhpCliMode()) {
exit;
}
if (!defined('PIWIK_ENABLE_ERROR_HANDLER') || PIWIK_ENABLE_ERROR_HANDLER) {
Piwik\ErrorHandler::registerErrorHandler();
Piwik\ExceptionHandler::setUp();
}
FrontController::setUpSafeMode();
$console = new Piwik\Console();
$console->run();

152
matomo/core/API/ApiRenderer.php Normal file
View file

@ -0,0 +1,152 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
use Exception;
use Piwik\Common;
use Piwik\DataTable\Renderer;
use Piwik\DataTable;
use Piwik\Piwik;
use Piwik\Plugin;
use Piwik\SettingsServer;
/**
* API renderer
*/
abstract class ApiRenderer
{
protected $request;
protected $hideIdSubDataTable;
final public function __construct($request)
{
$this->request = $request;
$this->init();
}
protected function init()
{
$this->hideIdSubDataTable = Common::getRequestVar('hideIdSubDatable', false, 'int', $this->request);
}
protected function shouldSendBacktrace()
{
return Common::isPhpCliMode() && SettingsServer::isArchivePhpTriggered();
}
abstract public function sendHeader();
public function renderSuccess($message)
{
return 'Success:' . $message;
}
/**
* @param $message
* @param Exception|\Throwable $exception
* @return mixed
*/
public function renderException($message, $exception)
{
return $message;
}
public function renderScalar($scalar)
{
$dataTable = new DataTable\Simple();
$dataTable->addRowsFromArray(array($scalar));
return $this->renderDataTable($dataTable);
}
public function renderDataTable($dataTable)
{
$renderer = $this->buildDataTableRenderer($dataTable);
return $renderer->render();
}
public function renderArray($array)
{
$renderer = $this->buildDataTableRenderer($array);
return $renderer->render();
}
public function renderObject($object)
{
$exception = new Exception('The API cannot handle this data structure.');
return $this->renderException($exception->getMessage(), $exception);
}
public function renderResource($resource)
{
$exception = new Exception('The API cannot handle this data structure.');
return $this->renderException($exception->getMessage(), $exception);
}
/**
* @param $dataTable
* @return Renderer
*/
protected function buildDataTableRenderer($dataTable)
{
$format = self::getFormatFromClass(get_class($this));
$idSite = Common::getRequestVar('idSite', 0, 'int', $this->request);
if (empty($idSite)) {
$idSite = 'all';
}
$renderer = Renderer::factory($format);
$renderer->setTable($dataTable);
$renderer->setIdSite($idSite);
$renderer->setRenderSubTables(Common::getRequestVar('expanded', false, 'int', $this->request));
$renderer->setHideIdSubDatableFromResponse($this->hideIdSubDataTable);
return $renderer;
}
/**
* @param string $format
* @param array $request
* @return ApiRenderer
* @throws Exception
*/
public static function factory($format, $request)
{
if (mb_strtolower($format) === 'json2') {
$format = 'json';
}
$formatToCheck = '\\' . ucfirst(strtolower($format));
$rendererClassnames = Plugin\Manager::getInstance()->findMultipleComponents('Renderer', 'Piwik\\API\\ApiRenderer');
foreach ($rendererClassnames as $klassName) {
if (Common::stringEndsWith($klassName, $formatToCheck)) {
return new $klassName($request);
}
}
$availableRenderers = array();
foreach ($rendererClassnames as $rendererClassname) {
$availableRenderers[] = self::getFormatFromClass($rendererClassname);
}
$availableRenderers = implode(', ', $availableRenderers);
Common::sendHeader('Content-Type: text/plain; charset=utf-8');
throw new Exception(Piwik::translate('General_ExceptionInvalidRendererFormat', array($format, $availableRenderers)));
}
private static function getFormatFromClass($klassname)
{
$klass = explode('\\', $klassname);
return strtolower(end($klass));
}
}

57
matomo/core/API/CORSHandler.php Normal file
View file

@ -0,0 +1,57 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
use Piwik\Common;
use Piwik\Url;
class CORSHandler
{
/**
* @var array
*/
protected $domains;
public function __construct()
{
$this->domains = Url::getCorsHostsFromConfig();
}
public function handle()
{
if (empty($this->domains)) {
return;
}
Common::sendHeader('Vary: Origin');
// allow Piwik to serve data to all domains
if (in_array("*", $this->domains)) {
Common::sendHeader('Access-Control-Allow-Credentials: true');
if (!empty($_SERVER['HTTP_ORIGIN'])) {
Common::sendHeader('Access-Control-Allow-Origin: ' . $_SERVER['HTTP_ORIGIN']);
return;
}
Common::sendHeader('Access-Control-Allow-Origin: *');
return;
}
// specifically allow if it is one of the allowlisted CORS domains
if (!empty($_SERVER['HTTP_ORIGIN'])) {
$origin = $_SERVER['HTTP_ORIGIN'];
if (in_array($origin, $this->domains, true)) {
Common::sendHeader('Access-Control-Allow-Credentials: true');
Common::sendHeader('Access-Control-Allow-Origin: ' . $_SERVER['HTTP_ORIGIN']);
}
}
}
}

245
matomo/core/API/DataTableGenericFilter.php Normal file
View file

@ -0,0 +1,245 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
use Exception;
use Piwik\Common;
use Piwik\DataTable;
use Piwik\Plugin\ProcessedMetric;
use Piwik\Plugin\Report;
class DataTableGenericFilter
{
/**
* List of filter names not to run.
*
* @var string[]
*/
private $disabledFilters = array();
/**
* @var Report
*/
private $report;
/**
* @var array
*/
private $request;
/**
* Constructor
*
* @param $request
*/
public function __construct($request, $report)
{
$this->request = $request;
$this->report = $report;
}
/**
* Filters the given data table
*
* @param DataTable $table
*/
public function filter($table)
{
$this->applyGenericFilters($table);
}
/**
* Makes sure a set of filters are not run.
*
* @param string[] $filterNames The name of each filter to disable.
*/
public function disableFilters($filterNames)
{
$this->disabledFilters = array_unique(array_merge($this->disabledFilters, $filterNames));
}
/**
* Returns an array containing the information of the generic Filter
* to be applied automatically to the data resulting from the API calls.
*
* Order to apply the filters:
* 1 - Filter that remove filtered rows
* 2 - Filter that sort the remaining rows
* 3 - Filter that keep only a subset of the results
* 4 - Presentation filters
*
* @return array See the code for spec
*/
public static function getGenericFiltersInformation()
{
return array(
array('Pattern',
array(
'filter_column' => array('string', 'label'),
'filter_pattern' => array('string')
)),
array('PatternRecursive',
array(
'filter_column_recursive' => array('string', 'label'),
'filter_pattern_recursive' => array('string'),
)),
array('ExcludeLowPopulation',
array(
'filter_excludelowpop' => array('string'),
'filter_excludelowpop_value' => array('float', '0'),
)),
array('Sort',
array(
'filter_sort_column' => array('string'),
'filter_sort_order' => array('string', 'desc'),
$naturalSort = true,
$recursiveSort = true,
'filter_sort_column_secondary' => true
)),
array('Truncate',
array(
'filter_truncate' => array('integer'),
)),
array('Limit',
array(
'filter_offset' => array('integer', '0'),
'filter_limit' => array('integer'),
'keep_summary_row' => array('integer', '0'),
))
);
}
private function getGenericFiltersHavingDefaultValues()
{
$filters = self::getGenericFiltersInformation();
if ($this->report && $this->report->getDefaultSortColumn()) {
foreach ($filters as $index => $filter) {
if ($filter[0] === 'Sort') {
$filters[$index][1]['filter_sort_column'] = array('string', $this->report->getDefaultSortColumn());
$filters[$index][1]['filter_sort_order'] = array('string', $this->report->getDefaultSortOrder());
$callback = $this->report->getSecondarySortColumnCallback();
if (is_callable($callback)) {
$filters[$index][1]['filter_sort_column_secondary'] = $callback;
}
}
}
}
return $filters;
}
/**
* Apply generic filters to the DataTable object resulting from the API Call.
* Disable this feature by setting the parameter disable_generic_filters to 1 in the API call request.
*
* @param DataTable $datatable
* @return bool
*/
protected function applyGenericFilters($datatable)
{
if ($datatable instanceof DataTable\Map) {
$tables = $datatable->getDataTables();
foreach ($tables as $table) {
$this->applyGenericFilters($table);
}
return;
}
$tableDisabledFilters = $datatable->getMetadata(DataTable::GENERIC_FILTERS_TO_DISABLE_METADATA_NAME) ?: [];
$genericFilters = $this->getGenericFiltersHavingDefaultValues();
$filterApplied = false;
foreach ($genericFilters as $filterMeta) {
$filterName = $filterMeta[0];
$filterParams = $filterMeta[1];
$filterParameters = array();
$exceptionRaised = false;
if (in_array($filterName, $this->disabledFilters)
|| in_array($filterName, $tableDisabledFilters)
) {
continue;
}
foreach ($filterParams as $name => $info) {
if (!is_array($info)) {
// hard coded value that cannot be changed via API, see eg $naturalSort = true in 'Sort'
$filterParameters[] = $info;
} else {
// parameter type to cast to
$type = $info[0];
// default value if specified, when the parameter doesn't have a value
$defaultValue = null;
if (isset($info[1])) {
$defaultValue = $info[1];
}
try {
$value = Common::getRequestVar($name, $defaultValue, $type, $this->request);
settype($value, $type);
$filterParameters[] = $value;
} catch (Exception $e) {
$exceptionRaised = true;
break;
}
}
}
if (!$exceptionRaised) {
$datatable->filter($filterName, $filterParameters);
$filterApplied = true;
}
}
return $filterApplied;
}
public function areProcessedMetricsNeededFor($metrics)
{
$columnQueryParameters = array(
'filter_column',
'filter_column_recursive',
'filter_excludelowpop',
'filter_sort_column'
);
foreach ($columnQueryParameters as $queryParamName) {
$queryParamValue = Common::getRequestVar($queryParamName, false, $type = null, $this->request);
if (!empty($queryParamValue)
&& $this->containsProcessedMetric($metrics, $queryParamValue)
) {
return true;
}
}
return false;
}
/**
* @param ProcessedMetric[] $metrics
* @param string $name
* @return bool
*/
private function containsProcessedMetric($metrics, $name)
{
foreach ($metrics as $metric) {
if ($metric instanceof ProcessedMetric
&& $metric->getName() == $name
) {
return true;
}
}
return false;
}
}

208
matomo/core/API/DataTableManipulator.php Normal file
View file

@ -0,0 +1,208 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
use Exception;
use Piwik\Archive\DataTableFactory;
use Piwik\Container\StaticContainer;
use Piwik\DataTable\Row;
use Piwik\DataTable;
use Piwik\Period\Range;
use Piwik\Plugins\API\API;
/**
* Base class for manipulating data tables.
* It provides generic mechanisms like iteration and loading subtables.
*
* The manipulators are used in ResponseBuilder and are triggered by
* API parameters. They are not filters because they don't work on the pre-
* fetched nested data tables. Instead, they load subtables using this base
* class. This way, they can only load the tables they really need instead
* of using expanded=1. Another difference between manipulators and filters
* is that filters keep the overall structure of the table intact while
* manipulators can change the entire thing.
*/
abstract class DataTableManipulator
{
protected $apiModule;
protected $apiMethod;
protected $request;
protected $apiMethodForSubtable;
/**
* Constructor
*
* @param bool $apiModule
* @param bool $apiMethod
* @param array $request
*/
public function __construct($apiModule = false, $apiMethod = false, $request = array())
{
$this->apiModule = $apiModule;
$this->apiMethod = $apiMethod;
$this->request = $request;
}
/**
* This method can be used by subclasses to iterate over data tables that might be
* data table maps. It calls back the template method self::doManipulate for each table.
* This way, data table arrays can be handled in a transparent fashion.
*
* @param DataTable\Map|DataTable $dataTable
* @throws Exception
* @return DataTable\Map|DataTable
*/
protected function manipulate($dataTable)
{
if ($dataTable instanceof DataTable\Map) {
return $this->manipulateDataTableMap($dataTable);
} elseif ($dataTable instanceof DataTable) {
return $this->manipulateDataTable($dataTable);
} else {
return $dataTable;
}
}
/**
* Manipulates child DataTables of a DataTable\Map. See @manipulate for more info.
*
* @param DataTable\Map $dataTable
* @return DataTable\Map
*/
protected function manipulateDataTableMap($dataTable)
{
$result = $dataTable->getEmptyClone();
foreach ($dataTable->getDataTables() as $tableLabel => $childTable) {
$newTable = $this->manipulate($childTable);
$result->addTable($newTable, $tableLabel);
}
return $result;
}
/**
* Manipulates a single DataTable instance. Derived classes must define
* this function.
*/
abstract protected function manipulateDataTable($dataTable);
/**
* Load the subtable for a row.
* Returns null if none is found.
*
* @param DataTable $dataTable
* @param Row $row
*
* @return DataTable
*/
protected function loadSubtable($dataTable, $row)
{
if (!($this->apiModule && $this->apiMethod && count($this->request))) {
return null;
}
$request = $this->request;
$idSubTable = $row->getIdSubDataTable();
if ($idSubTable === null) {
return null;
}
$request['idSubtable'] = $idSubTable;
if ($dataTable) {
$period = $dataTable->getMetadata(DataTableFactory::TABLE_METADATA_PERIOD_INDEX);
if ($period instanceof Range) {
$request['date'] = $period->getDateStart() . ',' . $period->getDateEnd();
} else {
$request['date'] = $period->getDateStart()->toString();
}
}
$method = $this->getApiMethodForSubtable($request);
return $this->callApiAndReturnDataTable($this->apiModule, $method, $request);
}
/**
* In this method, subclasses can clean up the request array for loading subtables
* in order to make ResponseBuilder behave correctly (e.g. not trigger the
* manipulator again).
*
* @param $request
* @return
*/
abstract protected function manipulateSubtableRequest($request);
/**
* Extract the API method for loading subtables from the meta data
*
* @throws Exception
* @return string
*/
protected function getApiMethodForSubtable($request)
{
if (!$this->apiMethodForSubtable) {
if (!empty($request['idSite'])) {
$idSite = $request['idSite'];
} else {
$idSite = 'all';
}
$apiParameters = array();
$entityNames = StaticContainer::get('entities.idNames');
foreach ($entityNames as $idName) {
if (!empty($request[$idName])) {
$apiParameters[$idName] = $request[$idName];
}
}
$meta = API::getInstance()->getMetadata($idSite, $this->apiModule, $this->apiMethod, $apiParameters);
if (empty($meta) && array_key_exists('idGoal', $apiParameters)) {
unset($apiParameters['idGoal']);
$meta = API::getInstance()->getMetadata($idSite, $this->apiModule, $this->apiMethod, $apiParameters);
}
if (empty($meta)) {
throw new Exception(sprintf(
"The DataTable cannot be manipulated: Metadata for report %s.%s could not be found. You can define the metadata in a hook, see example at: https://developer.matomo.org/api-reference/events#apigetreportmetadata",
$this->apiModule, $this->apiMethod
));
}
if (isset($meta[0]['actionToLoadSubTables'])) {
$this->apiMethodForSubtable = $meta[0]['actionToLoadSubTables'];
} else {
$this->apiMethodForSubtable = $this->apiMethod;
}
}
return $this->apiMethodForSubtable;
}
protected function callApiAndReturnDataTable($apiModule, $method, $request)
{
$class = Request::getClassNameAPI($apiModule);
$request = $this->manipulateSubtableRequest($request);
$request['serialize'] = 0;
$request['expanded'] = 0;
$request['format'] = 'original';
$request['format_metrics'] = 0;
$request['compare'] = 0;
// don't want to run recursive filters on the subtables as they are loaded,
// otherwise the result will be empty in places (or everywhere). instead we
// run it on the flattened table.
unset($request['filter_pattern_recursive']);
$dataTable = Proxy::getInstance()->call($class, $method, $request);
$response = new ResponseBuilder($format = 'original', $request);
$response->disableSendHeader();
$dataTable = $response->getResponse($dataTable, $apiModule, $method);
return $dataTable;
}
}

235
matomo/core/API/DataTableManipulator/Flattener.php Normal file
View file

@ -0,0 +1,235 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API\DataTableManipulator;
use Piwik\API\DataTableManipulator;
use Piwik\DataTable;
use Piwik\DataTable\Row;
use Piwik\Plugin\ReportsProvider;
/**
* This class is responsible for flattening data tables.
*
* It loads subtables and combines them into a single table by concatenating the labels.
* This manipulator is triggered by using flat=1 in the API request.
*/
class Flattener extends DataTableManipulator
{
private $includeAggregateRows = false;
/**
* If the flattener is used after calling this method, aggregate rows will
* be included in the result. This can be useful when they contain data that
* the leafs don't have (e.g. conversion stats in some cases).
*/
public function includeAggregateRows()
{
$this->includeAggregateRows = true;
}
/**
* Separator for building recursive labels (or paths)
* @var string
*/
public $recursiveLabelSeparator = '';
/**
* @param DataTable $dataTable
* @param string $recursiveLabelSeparator
* @return DataTable|DataTable\Map
*/
public function flatten($dataTable, $recursiveLabelSeparator)
{
$this->recursiveLabelSeparator = $recursiveLabelSeparator;
return $this->manipulate($dataTable);
}
/**
* Template method called from self::manipulate.
* Flatten each data table.
*
* @param DataTable $dataTable
* @return DataTable
*/
protected function manipulateDataTable($dataTable)
{
$newDataTable = $dataTable->getEmptyClone($keepFilters = true);
if ($dataTable->getTotalsRow()) {
$newDataTable->setTotalsRow($dataTable->getTotalsRow());
}
// this recursive filter will be applied to subtables
$dataTable->filter('ReplaceSummaryRowLabel');
$dataTable->filter('ReplaceColumnNames');
$report = ReportsProvider::factory($this->apiModule, $this->apiMethod);
if (!empty($report)) {
$dimension = $report->getDimension();
}
$dimensionName = !empty($dimension) ? str_replace('.', '_', $dimension->getId()) : 'label1';
$this->flattenDataTableInto($dataTable, $newDataTable, $level = 1, $dimensionName);
return $newDataTable;
}
/**
* @param $dataTable DataTable
* @param $newDataTable
* @param $dimensionName
*/
protected function flattenDataTableInto($dataTable, $newDataTable, $level, $dimensionName, $prefix = '', $logo = false)
{
foreach ($dataTable->getRows() as $rowId => $row) {
$this->flattenRow($row, $rowId, $newDataTable, $level, $dimensionName, $prefix, $logo);
}
}
/**
* @param Row $row
* @param DataTable $dataTable
* @param string $labelPrefix
* @param string $dimensionName
* @param bool $parentLogo
*/
private function flattenRow(Row $row, $rowId, DataTable $dataTable, $level, $dimensionName,
$labelPrefix = '', $parentLogo = false)
{
$dimensions = $dataTable->getMetadata('dimensions');
if (empty($dimensions)) {
$dimensions = [];
}
if (!in_array($dimensionName, $dimensions)) {
$dimensions[] = $dimensionName;
}
$dataTable->setMetadata('dimensions', $dimensions);
$origLabel = $label = $row->getColumn('label');
if ($label !== false) {
$origLabel = $label = trim($label);
if ($this->recursiveLabelSeparator == '/') {
if (substr($label, 0, 1) == '/' && substr($labelPrefix, -1) == '/') {
$origLabel = $label = substr($label, 1);
} elseif ($rowId === DataTable::ID_SUMMARY_ROW && $labelPrefix && $label != DataTable::LABEL_SUMMARY_ROW) {
$label = ' - ' . $label;
}
}
if ($rowId === DataTable::ID_SUMMARY_ROW) {
if ($row->getMetadata('url')) {
// remove url metadata for flattened summary rows
$row->deleteMetadata('url');
}
$row->setMetadata('is_summary', true);
}
$label = $labelPrefix . $label;
$row->setColumn('label', $label);
if ($row->getMetadata($dimensionName)) {
if ($rowId === DataTable::ID_SUMMARY_ROW && $this->recursiveLabelSeparator == '/') {
$origLabel = $row->getMetadata($dimensionName) . $this->recursiveLabelSeparator . ' - ' . $origLabel;
} else {
$origLabel = $row->getMetadata($dimensionName) . $this->recursiveLabelSeparator . $origLabel;
}
}
$row->setMetadata($dimensionName, $origLabel);
}
$logo = $row->getMetadata('logo');
if ($logo === false && $parentLogo !== false) {
$logo = $parentLogo;
$row->setMetadata('logo', $logo);
}
/** @var DataTable $subTable */
$subTable = $row->getSubtable();
if ($subTable) {
$subTable->applyQueuedFilters();
$row->deleteMetadata('idsubdatatable_in_db');
} else {
$subTable = $this->loadSubtable($dataTable, $row);
}
$row->removeSubtable();
if ($subTable === null) {
if ($this->includeAggregateRows) {
$row->setMetadata('is_aggregate', 0);
}
$dataTable->addRow($row);
} else {
if ($this->includeAggregateRows) {
$row->setMetadata('is_aggregate', 1);
$dataTable->addRow($row);
}
$prefix = $label . $this->recursiveLabelSeparator;
$report = ReportsProvider::factory($this->apiModule, $this->apiMethod);
if (!empty($report)) {
$subDimension = $report->getSubtableDimension();
}
if ($level === 2) {
$subDimension = $report->getThirdLeveltableDimension();
}
if (empty($subDimension)) {
$report = ReportsProvider::factory($this->apiModule, $this->getApiMethodForSubtable($this->request));
$subDimension = $report->getDimension();
}
$subDimensionName = $subDimension ? str_replace('.', '_', $subDimension->getId()) : 'label' . (substr_count($prefix, $this->recursiveLabelSeparator) + 1);
if ($origLabel !== false) {
foreach ($subTable->getRows() as $subRow) {
foreach ($row->getMetadata() as $name => $value) {
// do not set 'segment' parameter if there is a segmentValue on the row, since that will prevent the segmentValue
// from being used in DataTablePostProcessor
if ($name == 'segment' && $subRow->getMetadata('segmentValue') !== false) {
continue;
}
if ($subRow->getMetadata($name) === false) {
$subRow->setMetadata($name, $value);
}
}
$subRow->setMetadata($dimensionName, $origLabel);
}
}
$this->flattenDataTableInto($subTable, $dataTable, $level + 1, $subDimensionName, $prefix, $logo);
}
}
/**
* Remove the flat parameter from the subtable request
*
* @param array $request
* @return array
*/
protected function manipulateSubtableRequest($request)
{
unset($request['flat']);
return $request;
}
}

222
matomo/core/API/DataTableManipulator/LabelFilter.php Normal file
View file

@ -0,0 +1,222 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API\DataTableManipulator;
use Piwik\API\DataTableManipulator;
use Piwik\Common;
use Piwik\DataTable;
use Piwik\DataTable\Row;
/**
* This class is responsible for handling the label parameter that can be
* added to every API call. If the parameter is set, only the row with the matching
* label is returned.
*
* The labels passed to this class should be urlencoded.
* Some reports use recursive labels (e.g. action reports). Use > to join them.
*/
class LabelFilter extends DataTableManipulator
{
const SEPARATOR_RECURSIVE_LABEL = '>';
const TERMINAL_OPERATOR = '@';
private $labels;
private $addLabelIndex;
private $isComparing;
private $labelSeries;
const FLAG_IS_ROW_EVOLUTION = 'label_index';
/**
* Filter a data table by label.
* The filtered table is returned, which might be a new instance.
*
* $apiModule, $apiMethod and $request are needed load sub-datatables
* for the recursive search. If the label is not recursive, these parameters
* are not needed.
*
* @param string $labels the labels to search for
* @param DataTable $dataTable the data table to be filtered
* @param bool $addLabelIndex Whether to add label_index metadata describing which
* label a row corresponds to.
* @return DataTable
*/
public function filter($labels, $dataTable, $addLabelIndex = false)
{
if (!is_array($labels)) {
$labels = array($labels);
}
$this->labels = array_values($labels);
$this->addLabelIndex = (bool)$addLabelIndex;
$this->isComparing = $this->isComparing();
$labelSeries = Common::getRequestVar('labelSeries', '', 'string', $this->request);
$labelSeries = explode(',', $labelSeries);
$labelSeries = array_filter($labelSeries, 'strlen');
$this->labelSeries = $labelSeries;
$result = $this->manipulate($dataTable);
return $result;
}
/**
* Method for the recursive descend
*
* @param array $labelParts
* @param DataTable $dataTable
* @return Row|bool
*/
private function doFilterRecursiveDescend($labelParts, $dataTable)
{
// we need to make sure to rebuild the index as some filters change the label column directly via
// $row->setColumn('label', '') which would not be noticed in the label index otherwise.
$dataTable->rebuildIndex();
// search for the first part of the tree search
$labelPart = array_shift($labelParts);
$row = false;
foreach ($this->getLabelVariations($labelPart) as $labelPart) {
$row = $dataTable->getRowFromLabel($labelPart);
if ($row !== false) {
break;
}
}
if ($row === false) {
// not found
return false;
}
// end of tree search reached
if (count($labelParts) == 0) {
return $row;
}
$subTable = $this->loadSubtable($dataTable, $row);
if ($subTable === null) {
// no more subtables but label parts left => no match found
return false;
}
return $this->doFilterRecursiveDescend($labelParts, $subTable);
}
/**
* Clean up request for ResponseBuilder to behave correctly
*
* @param $request
*/
protected function manipulateSubtableRequest($request)
{
unset($request['label']);
unset($request['flat']);
$request['totals'] = 0;
$request['filter_sort_column'] = ''; // do not sort, we only want to find a matching column
return $request;
}
/**
* Use variations of the label to make it easier to specify the desired label
*
* Note: The HTML Encoded version must be tried first, since in ResponseBuilder the $label is unsanitized
* via Common::unsanitizeLabelParameter.
*
* @param string $originalLabel
* @return array
*/
private function getLabelVariations($originalLabel)
{
static $pageTitleReports = array('getPageTitles', 'getEntryPageTitles', 'getExitPageTitles');
$originalLabel = trim($originalLabel);
$isTerminal = substr($originalLabel, 0, 1) == self::TERMINAL_OPERATOR;
if ($isTerminal) {
$originalLabel = substr($originalLabel, 1);
}
$variations = array();
$label = trim(urldecode($originalLabel));
$sanitizedLabel = Common::sanitizeInputValue($label);
$variations[] = $sanitizedLabel;
if ($this->apiModule == 'Actions'
&& in_array($this->apiMethod, $pageTitleReports)
) {
if ($isTerminal) {
array_unshift($variations, ' ' . $sanitizedLabel);
array_unshift($variations, ' ' . $label);
} else {
// special case: the Actions.getPageTitles report prefixes some labels with a blank.
// the blank might be passed by the user but is removed in Request::getRequestArrayFromString.
$variations[] = ' ' . $sanitizedLabel;
$variations[] = ' ' . $label;
}
}
$variations[] = $label;
$variations = array_unique($variations);
return $variations;
}
/**
* Filter a DataTable instance. See @filter for more info.
*
* @param DataTable\Simple|DataTable\Map $dataTable
* @return mixed
*/
protected function manipulateDataTable($dataTable)
{
$result = $dataTable->getEmptyClone();
foreach ($this->labels as $labelIndex => $label) {
$row = null;
foreach ($this->getLabelVariations($label) as $labelVariation) {
$labelVariation = explode(self::SEPARATOR_RECURSIVE_LABEL, $labelVariation);
$row = $this->doFilterRecursiveDescend($labelVariation, $dataTable);
if ($row) {
if ($this->isComparing
&& isset($this->labelSeries[$labelIndex])
) {
$comparisons = $row->getComparisons();
if (!empty($comparisons)) {
$labelSeriesIndex = $this->labelSeries[$labelIndex];
$originalLabel = $row->getColumn('label');
$row = $comparisons->getRowFromId($labelSeriesIndex);
// add label and make sure it is the first column
$columns = array_merge(['label' => $originalLabel . ' ' . $row->getMetadata('compareSeriesPretty')], $row->getColumns());
$row->setColumns($columns);
}
}
if ($this->addLabelIndex) {
$row->setMetadata(self::FLAG_IS_ROW_EVOLUTION, $labelIndex);
}
$result->addRow($row);
break;
}
}
}
return $result;
}
private function isComparing()
{
return Common::getRequestVar('compare', 0, 'int', $this->request) == 1;
}
}

263
matomo/core/API/DataTableManipulator/ReportTotalsCalculator.php Normal file
View file

@ -0,0 +1,263 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API\DataTableManipulator;
use Piwik\API\DataTableManipulator;
use Piwik\API\DataTablePostProcessor;
use Piwik\Common;
use Piwik\DataTable;
use Piwik\Period;
use Piwik\Piwik;
use Piwik\Plugin\Report;
use Piwik\Plugin\ReportsProvider;
/**
* This class is responsible for setting the metadata property 'totals' on each dataTable if the report
* has a dimension. 'Totals' means it tries to calculate the total report value for each metric. For each
* the total number of visits, actions, ... for a given report / dataTable.
*/
class ReportTotalsCalculator extends DataTableManipulator
{
/**
* @var Report
*/
private $report;
/**
* Constructor
*
* @param bool $apiModule
* @param bool $apiMethod
* @param array $request
* @param Report $report
*/
public function __construct($apiModule = false, $apiMethod = false, $request = array(), $report = null)
{
parent::__construct($apiModule, $apiMethod, $request);
$this->report = $report;
}
/**
* @param DataTable $table
* @return \Piwik\DataTable|\Piwik\DataTable\Map
*/
public function calculate($table)
{
// apiModule and/or apiMethod is empty for instance in case when flat=1 is called. Basically whenever a
// datamanipulator calls the API and wants the dataTable in return, see callApiAndReturnDataTable().
// it is also not set for some settings API request etc.
if (empty($this->apiModule) || empty($this->apiMethod)) {
return $table;
}
try {
return $this->manipulate($table);
} catch (\Exception $e) {
// eg. requests with idSubtable may trigger this exception
// (where idSubtable was removed in
// ?module=API&method=Events.getNameFromCategoryId&idSubtable=1&secondaryDimension=eventName&format=XML&idSite=1&period=day&date=yesterday&flat=0
return $table;
}
}
/**
* Adds ratio metrics if possible.
*
* @param DataTable $dataTable
* @return DataTable
*/
protected function manipulateDataTable($dataTable)
{
if (!empty($this->report) && !$this->report->getDimension() && !$this->isAllMetricsReport()) {
// we currently do not calculate the total value for reports having no dimension
return $dataTable;
}
if (1 != Common::getRequestVar('totals', 1, 'integer', $this->request)) {
return $dataTable;
}
$firstLevelTable = $this->makeSureToWorkOnFirstLevelDataTable($dataTable);
if (!$firstLevelTable->getRowsCount()
|| $dataTable->getTotalsRow()
|| $dataTable->getMetadata('totals')
) {
return $dataTable;
}
// keeping queued filters would not only add various metadata but also break the totals calculator for some reports
// eg when needed metadata is missing to get site information (multisites.getall) etc
$clone = $firstLevelTable->getEmptyClone($keepFilters = false);
foreach ($firstLevelTable->getQueuedFilters() as $queuedFilter) {
if (is_array($queuedFilter) && 'ReplaceColumnNames' === $queuedFilter['className']) {
$clone->queueFilter($queuedFilter['className'], $queuedFilter['parameters']);
}
}
$tableMeta = $firstLevelTable->getMetadata(DataTable::COLUMN_AGGREGATION_OPS_METADATA_NAME);
/** @var DataTable\Row $totalRow */
$totalRow = null;
foreach ($firstLevelTable->getRows() as $row) {
if (!isset($totalRow)) {
$columns = $row->getColumns();
$columns['label'] = DataTable::LABEL_TOTALS_ROW;
$totalRow = new DataTable\Row(array(DataTable\Row::COLUMNS => $columns));
} else {
$totalRow->sumRow($row, $copyMetadata = false, $tableMeta);
}
}
$clone->addRow($totalRow);
if ($this->report
&& $this->report->getProcessedMetrics()
&& array_keys($this->report->getProcessedMetrics()) === array('nb_actions_per_visit', 'avg_time_on_site', 'bounce_rate', 'conversion_rate')) {
// hack for AllColumns table or default processed metrics
$clone->filter('AddColumnsProcessedMetrics', array($deleteRowsWithNoVisit = false));
}
$processor = new DataTablePostProcessor($this->apiModule, $this->apiMethod, $this->request);
$processor->applyComputeProcessedMetrics($clone);
$clone = $processor->applyQueuedFilters($clone);
$totalRowUnformatted = null;
foreach ($clone->getRows() as $row) {
/** * @var DataTable\Row $row */
if ($row->getColumn('label') === DataTable::LABEL_TOTALS_ROW) {
$totalRowUnformatted = $row->getColumns();
break;
}
}
$clone = $processor->applyMetricsFormatting($clone);
$totalRow = null;
foreach ($clone->getRows() as $row) {
/** * @var DataTable\Row $row */
if ($row->getColumn('label') === DataTable::LABEL_TOTALS_ROW) {
$totalRow = $row;
break;
}
}
if (!isset($totalRow) && $clone->getRowsCount() === 1) {
// if for some reason the processor renamed the totals row,
$totalRow = $clone->getFirstRow();
}
if (isset($totalRow)) {
$totals = $row->getColumns();
unset($totals['label']);
$dataTable->setMetadata('totals', $totals);
if (isset($totalRowUnformatted)) {
unset($totalRowUnformatted['label']);
$dataTable->setMetadata('totalsUnformatted', $totalRowUnformatted);
}
if (1 === Common::getRequestVar('keep_totals_row', 0, 'integer', $this->request)) {
$totalLabel = Common::getRequestVar('keep_totals_row_label', Piwik::translate('General_Totals'), 'string', $this->request);
$row->deleteMetadata(false);
$row->setColumn('label', $totalLabel);
$dataTable->setTotalsRow($row);
}
}
return $dataTable;
}
private function makeSureToWorkOnFirstLevelDataTable($table)
{
if (!array_key_exists('idSubtable', $this->request)) {
return $table;
}
$firstLevelReport = $this->findFirstLevelReport();
if (empty($firstLevelReport)) {
// it is not a subtable report
$module = $this->apiModule;
$action = $this->apiMethod;
} else {
$module = $firstLevelReport->getModule();
$action = $firstLevelReport->getAction();
}
$request = $this->request;
unset($request['idSubtable']); // to make sure we work on first level table
/** @var \Piwik\Period $period */
$period = $table->getMetadata('period');
if (!empty($period)) {
// we want a dataTable, not a dataTable\map
if (Period::isMultiplePeriod($request['date'], $request['period']) || 'range' == $period->getLabel()) {
$request['date'] = $period->getRangeString();
$request['period'] = 'range';
} else {
$request['date'] = $period->getDateStart()->toString();
$request['period'] = $period->getLabel();
}
}
$table = $this->callApiAndReturnDataTable($module, $action, $request);
if ($table instanceof DataTable\Map) {
$table = $table->mergeChildren();
}
return $table;
}
/**
* Make sure to get all rows of the first level table.
*
* @param array $request
* @return array
*/
protected function manipulateSubtableRequest($request)
{
$request['totals'] = 0;
$request['expanded'] = 0;
$request['filter_limit'] = -1;
$request['filter_offset'] = 0;
$request['filter_sort_column'] = '';
$parametersToRemove = array('flat');
if (!array_key_exists('idSubtable', $this->request)) {
$parametersToRemove[] = 'idSubtable';
}
foreach ($parametersToRemove as $param) {
if (array_key_exists($param, $request)) {
unset($request[$param]);
}
}
return $request;
}
private function findFirstLevelReport()
{
$reports = new ReportsProvider();
foreach ($reports->getAllReports() as $report) {
$actionToLoadSubtables = $report->getActionToLoadSubTables();
if ($actionToLoadSubtables == $this->apiMethod
&& $this->apiModule == $report->getModule()
) {
return $report;
}
}
return null;
}
private function isAllMetricsReport()
{
return $this->report->getModule() == 'API' && $this->report->getAction() == 'get';
}
}

509
matomo/core/API/DataTablePostProcessor.php Normal file
View file

@ -0,0 +1,509 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\API;
use Exception;
use Piwik\API\DataTableManipulator\Flattener;
use Piwik\API\DataTableManipulator\LabelFilter;
use Piwik\API\DataTableManipulator\ReportTotalsCalculator;
use Piwik\Common;
use Piwik\DataTable;
use Piwik\DataTable\DataTableInterface;
use Piwik\DataTable\Filter\PivotByDimension;
use Piwik\Metrics\Formatter;
use Piwik\Piwik;
use Piwik\Plugin\ProcessedMetric;
use Piwik\Plugin\Report;
use Piwik\Plugin\ReportsProvider;
use Piwik\Plugins\API\Filter\DataComparisonFilter;
use Piwik\Plugins\CoreHome\Columns\Metrics\EvolutionMetric;
/**
* Processes DataTables that should be served through Piwik's APIs. This processing handles
* special query parameters and computes processed metrics. It does not included rendering to
* output formats (eg, 'xml').
*/
class DataTablePostProcessor
{
const PROCESSED_METRICS_COMPUTED_FLAG = 'processed_metrics_computed';
/**
* @var null|Report
*/
private $report;
/**
* @var string[]
*/
private $request;
/**
* @var string
*/
private $apiModule;
/**
* @var string
*/
private $apiMethod;
/**
* @var Inconsistencies
*/
private $apiInconsistencies;
/**
* @var Formatter
*/
private $formatter;
private $callbackBeforeGenericFilters;
private $callbackAfterGenericFilters;
/**
* Constructor.
*/
public function __construct($apiModule, $apiMethod, $request)
{
$this->apiModule = $apiModule;
$this->apiMethod = $apiMethod;
$this->setRequest($request);
$this->report = ReportsProvider::factory($apiModule, $apiMethod);
$this->apiInconsistencies = new Inconsistencies();
$this->setFormatter(new Formatter());
}
public function setFormatter(Formatter $formatter)
{
$this->formatter = $formatter;
}
public function setRequest($request)
{
$this->request = $request;
}
public function setCallbackBeforeGenericFilters($callbackBeforeGenericFilters)
{
$this->callbackBeforeGenericFilters = $callbackBeforeGenericFilters;
}
public function setCallbackAfterGenericFilters($callbackAfterGenericFilters)
{
$this->callbackAfterGenericFilters = $callbackAfterGenericFilters;
}
/**
* Apply post-processing logic to a DataTable of a report for an API request.
*
* @param DataTableInterface $dataTable The data table to process.
* @return DataTableInterface A new data table.
*/
public function process(DataTableInterface $dataTable)
{
// TODO: when calculating metrics before hand, only calculate for needed metrics, not all. NOTE:
// this is non-trivial since it will require, eg, to make sure processed metrics aren't added
// after pivotBy is handled.
$dataTable = $this->applyPivotByFilter($dataTable);
$dataTable = $this->applyTotalsCalculator($dataTable);
$dataTable = $this->applyFlattener($dataTable);
if ($this->callbackBeforeGenericFilters) {
call_user_func($this->callbackBeforeGenericFilters, $dataTable);
}
$dataTable = $this->applyGenericFilters($dataTable);
$this->applyComputeProcessedMetrics($dataTable);
$dataTable = $this->applyComparison($dataTable);
if ($this->callbackAfterGenericFilters) {
call_user_func($this->callbackAfterGenericFilters, $dataTable);
}
// we automatically safe decode all datatable labels (against xss)
$dataTable->queueFilter('SafeDecodeLabel');
$dataTable = $this->convertSegmentValueToSegment($dataTable);
$dataTable = $this->applyQueuedFilters($dataTable);
$dataTable = $this->applyRequestedColumnDeletion($dataTable);
$dataTable = $this->applyLabelFilter($dataTable);
$dataTable = $this->applyMetricsFormatting($dataTable);
return $dataTable;
}
private function convertSegmentValueToSegment(DataTableInterface $dataTable)
{
$dataTable->filter('AddSegmentBySegmentValue', array($this->report));
$dataTable->filter('ColumnCallbackDeleteMetadata', array('segmentValue'));
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyPivotByFilter(DataTableInterface $dataTable)
{
$pivotBy = Common::getRequestVar('pivotBy', false, 'string', $this->request);
if (!empty($pivotBy)) {
$this->applyComputeProcessedMetrics($dataTable);
$dataTable = $this->convertSegmentValueToSegment($dataTable);
$pivotByColumn = Common::getRequestVar('pivotByColumn', false, 'string', $this->request);
$pivotByColumnLimit = Common::getRequestVar('pivotByColumnLimit', false, 'int', $this->request);
$dataTable->filter('PivotByDimension', array($this->report, $pivotBy, $pivotByColumn, $pivotByColumnLimit,
PivotByDimension::isSegmentFetchingEnabledInConfig()));
$dataTable->filter('ColumnCallbackDeleteMetadata', array('segment'));
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTable|DataTableInterface|DataTable\Map
*/
public function applyFlattener($dataTable)
{
if (Common::getRequestVar('flat', '0', 'string', $this->request) == '1') {
// skip flattening if not supported by report and remove subtables only
if ($this->report && !$this->report->supportsFlatten()) {
$dataTable->filter('RemoveSubtables');
return $dataTable;
}
$flattener = new Flattener($this->apiModule, $this->apiMethod, $this->request);
if (Common::getRequestVar('include_aggregate_rows', '0', 'string', $this->request) == '1') {
$flattener->includeAggregateRows();
}
$recursiveLabelSeparator = ' - ';
if ($this->report) {
$recursiveLabelSeparator = $this->report->getRecursiveLabelSeparator();
}
$dataTable = $flattener->flatten($dataTable, $recursiveLabelSeparator);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyTotalsCalculator($dataTable)
{
if (1 == Common::getRequestVar('totals', '1', 'integer', $this->request)) {
$calculator = new ReportTotalsCalculator($this->apiModule, $this->apiMethod, $this->request, $this->report);
$dataTable = $calculator->calculate($dataTable);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyGenericFilters($dataTable)
{
// if the flag disable_generic_filters is defined we skip the generic filters
if (0 == Common::getRequestVar('disable_generic_filters', '0', 'string', $this->request)) {
$this->applyProcessedMetricsGenericFilters($dataTable);
$genericFilter = new DataTableGenericFilter($this->request, $this->report);
$self = $this;
$report = $this->report;
$dataTable->filter(function (DataTable $table) use ($genericFilter, $report, $self) {
$processedMetrics = Report::getProcessedMetricsForTable($table, $report);
if ($genericFilter->areProcessedMetricsNeededFor($processedMetrics)) {
$self->computeProcessedMetrics($table);
}
});
$label = self::getLabelFromRequest($this->request);
if (!empty($label)) {
$genericFilter->disableFilters(array('Limit', 'Truncate'));
}
$genericFilter->filter($dataTable);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyProcessedMetricsGenericFilters($dataTable)
{
$addNormalProcessedMetrics = null;
try {
$addNormalProcessedMetrics = Common::getRequestVar(
'filter_add_columns_when_show_all_columns', null, 'integer', $this->request);
} catch (Exception $ex) {
// ignore
}
if ($addNormalProcessedMetrics !== null) {
$dataTable->filter('AddColumnsProcessedMetrics', array($addNormalProcessedMetrics));
}
$addGoalProcessedMetrics = null;
try {
$addGoalProcessedMetrics = Common::getRequestVar(
'filter_update_columns_when_show_all_goals', false, 'string', $this->request);
if ((int) $addGoalProcessedMetrics === 0
&& $addGoalProcessedMetrics !== '0'
&& $addGoalProcessedMetrics != Piwik::LABEL_ID_GOAL_IS_ECOMMERCE_ORDER
&& $addGoalProcessedMetrics != Piwik::LABEL_ID_GOAL_IS_ECOMMERCE_CART
) {
$addGoalProcessedMetrics = null;
}
} catch (Exception $ex) {
// ignore
}
$goalsToProcess = null;
try {
$goalsToProcess = Common::getRequestVar('filter_show_goal_columns_process_goals', null, 'string', $this->request);
$goalsToProcess = explode(',', $goalsToProcess);
$goalsToProcess = array_map('trim', $goalsToProcess);
$goalsToProcess = array_filter($goalsToProcess);
} catch (Exception $ex) {
// ignore
}
if ($addGoalProcessedMetrics !== null) {
$idGoal = Common::getRequestVar(
'idGoal', DataTable\Filter\AddColumnsProcessedMetricsGoal::GOALS_OVERVIEW, 'string', $this->request);
$dataTable->filter('AddColumnsProcessedMetricsGoal', array($ignore = true, $idGoal, $goalsToProcess));
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyQueuedFilters($dataTable)
{
// if the flag disable_queued_filters is defined we skip the filters that were queued
if (Common::getRequestVar('disable_queued_filters', 0, 'int', $this->request) == 0) {
$dataTable->applyQueuedFilters();
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyRequestedColumnDeletion($dataTable)
{
// use the ColumnDelete filter if hideColumns/showColumns is provided (must be done
// after queued filters are run so processed metrics can be removed, too)
$hideColumns = Common::getRequestVar('hideColumns', '', 'string', $this->request);
$showColumns = Common::getRequestVar('showColumns', '', 'string', $this->request);
$hideColumnsRecursively = Common::getRequestVar('hideColumnsRecursively', intval($this->report && $this->report->getModule() == 'Live'), 'int', $this->request);
$showRawMetrics = Common::getRequestVar('showRawMetrics', 0, 'int', $this->request);
if (!empty($hideColumns)
|| !empty($showColumns)
) {
$dataTable->filter('ColumnDelete', array($hideColumns, $showColumns, $deleteIfZeroOnly = false, $hideColumnsRecursively));
} else if ($showRawMetrics !== 1) {
$this->removeTemporaryMetrics($dataTable);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
*/
public function removeTemporaryMetrics(DataTableInterface $dataTable)
{
$allColumns = !empty($this->report) ? $this->report->getAllMetrics() : array();
$report = $this->report;
$dataTable->filter(function (DataTable $table) use ($report, $allColumns) {
$processedMetrics = Report::getProcessedMetricsForTable($table, $report);
$allTemporaryMetrics = array();
foreach ($processedMetrics as $metric) {
$allTemporaryMetrics = array_merge($allTemporaryMetrics, $metric->getTemporaryMetrics());
}
if (!empty($allTemporaryMetrics)) {
$table->filter('ColumnDelete', array($allTemporaryMetrics));
}
});
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyLabelFilter($dataTable)
{
$label = self::getLabelFromRequest($this->request);
// apply label filter: only return rows matching the label parameter (more than one if more than one label)
if (!empty($label)) {
$addLabelIndex = Common::getRequestVar('labelFilterAddLabelIndex', 0, 'int', $this->request) == 1;
$filter = new LabelFilter($this->apiModule, $this->apiMethod, $this->request);
$dataTable = $filter->filter($label, $dataTable, $addLabelIndex);
}
return $dataTable;
}
/**
* @param DataTableInterface $dataTable
* @return DataTableInterface
*/
public function applyMetricsFormatting($dataTable)
{
$formatMetrics = Common::getRequestVar('format_metrics', 0, 'string', $this->request);
if ($formatMetrics == '0') {
return $dataTable;
}
// in Piwik 2.X & below, metrics are not formatted in API responses except for percents.
// this code implements this inconsistency
$onlyFormatPercents = $formatMetrics === 'bc';
$metricsToFormat = null;
if ($onlyFormatPercents) {
$metricsToFormat = $this->apiInconsistencies->getPercentMetricsToFormat();
}
// 'all' is a special value that indicates we should format non-processed metrics that are identified
// by string, like 'revenue'. this should be removed when all metrics are using the `Metric` class.
$formatAll = $formatMetrics === 'all';
$dataTable->filter(array($this->formatter, 'formatMetrics'), array($this->report, $metricsToFormat, $formatAll));
return $dataTable;
}
/**
* Returns the value for the label query parameter which can be either a string
* (ie, label=...) or array (ie, label[]=...).
*
* @param array $request
* @return array
*/
public static function getLabelFromRequest($request)
{
$label = Common::getRequestVar('label', array(), 'array', $request);
if (empty($label)) {
$label = Common::getRequestVar('label', '', 'string', $request);
if (!empty($label)) {
$label = array($label);
}
}
$label = self::unsanitizeLabelParameter($label);
return $label;
}
public static function unsanitizeLabelParameter($label)
{
// this is needed because Proxy uses Common::getRequestVar which in turn
// uses Common::sanitizeInputValue. This causes the > that separates recursive labels
// to become &gt; and we need to undo that here.
$label = str_replace( htmlentities('>', ENT_COMPAT | ENT_HTML401, 'UTF-8'), '>', $label);
return $label;
}
public function computeProcessedMetrics(DataTable $dataTable)
{
if ($dataTable->getMetadata(self::PROCESSED_METRICS_COMPUTED_FLAG)) {
return;
}
/** @var ProcessedMetric[] $processedMetrics */
$processedMetrics = Report::getProcessedMetricsForTable($dataTable, $this->report);
if (empty($processedMetrics)) {
return;
}
$dataTable->setMetadata(self::PROCESSED_METRICS_COMPUTED_FLAG, true);
foreach ($processedMetrics as $name => $processedMetric) {
if (!$processedMetric->beforeCompute($this->report, $dataTable)) {
continue;
}
foreach ($dataTable->getRows() as $row) {
if ($row->getColumn($name) !== false) { // only compute the metric if it has not been computed already
continue;
}
$computedValue = $processedMetric->compute($row);
if ($computedValue !== false) {
$row->addColumn($name, $computedValue);
// Add a trend column for evolution metrics
if ($processedMetric instanceof EvolutionMetric) {
$row->addColumn($processedMetric->getTrendName(), $processedMetric->getTrendValue($computedValue));
}
}
}
}
foreach ($dataTable->getRows() as $row) {
$subtable = $row->getSubtable();
if (!empty($subtable)) {
foreach ($processedMetrics as $name => $processedMetric) {
$processedMetric->beforeComputeSubtable($row);
}
$this->computeProcessedMetrics($subtable);
foreach ($processedMetrics as $name => $processedMetric) {
$processedMetric->afterComputeSubtable($row);
}
}
}
}
public function applyComputeProcessedMetrics(DataTableInterface $dataTable)
{
$dataTable->filter(array($this, 'computeProcessedMetrics'));
}
public function applyComparison(DataTableInterface $dataTable)
{
$compare = Common::getRequestVar('compare', '0', 'int', $this->request);
if ($compare != 1) {
return $dataTable;
}
$filter = new DataComparisonFilter($this->request, $this->report);
$filter->compare($dataTable);
$dataTable->filter(function (DataTable $table) {
foreach ($table->getRows() as $row) {
$comparisons = $row->getComparisons();
if (!empty($comparisons)) {
$this->computeProcessedMetrics($comparisons);
}
}
});
return $dataTable;
}
}

401
matomo/core/API/DocumentationGenerator.php Normal file
View file

@ -0,0 +1,401 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
use Exception;
use Piwik\Common;
use Piwik\Container\StaticContainer;
use Piwik\Piwik;
use Piwik\Url;
use ReflectionClass;
/**
* Possible tags to use in APIs
*
* @hide -> Won't be shown in list of all APIs but is also not possible to be called via HTTP API
* @hideForAll Same as @hide
* @hideExceptForSuperUser Same as @hide but still shown and possible to be called by a user with super user access
* @internal -> Won't be shown in list of all APIs but is possible to be called via HTTP API
*/
class DocumentationGenerator
{
protected $countPluginsLoaded = 0;
/**
* trigger loading all plugins with an API.php file in the Proxy
*/
public function __construct()
{
$plugins = \Piwik\Plugin\Manager::getInstance()->getLoadedPluginsName();
foreach ($plugins as $plugin) {
try {
$className = Request::getClassNameAPI($plugin);
Proxy::getInstance()->registerClass($className);
} catch (Exception $e) {
}
}
}
/**
* Returns a HTML page containing help for all the successfully loaded APIs.
*
* @param bool $outputExampleUrls
* @return string
*/
public function getApiDocumentationAsString($outputExampleUrls = true)
{
list($toc, $str) = $this->generateDocumentation($outputExampleUrls, $prefixUrls = '', $displayTitlesAsAngularDirective = true);
return "<div piwik-content-block content-title='Quick access to APIs' id='topApiRef' name='topApiRef'>
$toc</div>
$str";
}
/**
* Used on developer.piwik.org
*
* @param bool|true $outputExampleUrls
* @param string $prefixUrls
* @return string
*/
public function getApiDocumentationAsStringForDeveloperReference($outputExampleUrls = true, $prefixUrls = '')
{
list($toc, $str) = $this->generateDocumentation($outputExampleUrls, $prefixUrls, $displayTitlesAsAngularDirective = false);
return "<h2 id='topApiRef' name='topApiRef'>Quick access to APIs</h2>
$toc
$str";
}
protected function prepareModuleToDisplay($moduleName)
{
return "<a href='#$moduleName'>$moduleName</a><br/>";
}
protected function prepareMethodToDisplay($moduleName, $info, $methods, $class, $outputExampleUrls, $prefixUrls, $displayTitlesAsAngularDirective)
{
$str = '';
$str .= "\n<a name='$moduleName' id='$moduleName'></a>";
if($displayTitlesAsAngularDirective) {
$str .= "<div piwik-content-block content-title='Module " . $moduleName . "'>";
} else {
$str .= "<h2>Module " . $moduleName . "</h2>";
}
$info['__documentation'] = $this->checkDocumentation($info['__documentation']);
$str .= "<div class='apiDescription'> " . $info['__documentation'] . " </div>";
foreach ($methods as $methodName) {
if (Proxy::getInstance()->isDeprecatedMethod($class, $methodName)) {
continue;
}
$params = $this->getParametersString($class, $methodName);
$str .= "\n <div class='apiMethod'>- <b>$moduleName.$methodName </b>" . $params . "";
$str .= '<small>';
if ($outputExampleUrls) {
$str .= $this->addExamples($class, $methodName, $prefixUrls);
}
$str .= '</small>';
$str .= "</div>\n";
}
if($displayTitlesAsAngularDirective) {
$str .= "</div>";
}
return $str;
}
protected function prepareModulesAndMethods($info, $moduleName)
{
$toDisplay = array();
foreach ($info as $methodName => $infoMethod) {
if ($methodName == '__documentation') {
continue;
}
$toDisplay[$moduleName][] = $methodName;
}
return $toDisplay;
}
protected function addExamples($class, $methodName, $prefixUrls)
{
$token = Piwik::getCurrentUserTokenAuth();
$token_auth_url = "&token_auth=" . $token;
if ($token !== 'anonymous') {
$token_auth_url .= "&force_api_session=1";
}
$parametersToSet = array(
'idSite' => Common::getRequestVar('idSite', 1, 'int'),
'period' => Common::getRequestVar('period', 'day', 'string'),
'date' => Common::getRequestVar('date', 'today', 'string')
);
$str = '';
$str .= "<span class=\"example\">";
$exampleUrl = $this->getExampleUrl($class, $methodName, $parametersToSet);
if ($exampleUrl !== false) {
$lastNUrls = '';
if (preg_match('/(&period)|(&date)/', $exampleUrl)) {
$exampleUrlRss = $prefixUrls . $this->getExampleUrl($class, $methodName, array('date' => 'last10', 'period' => 'day') + $parametersToSet);
$lastNUrls = ", RSS of the last <a target='_blank' href='$exampleUrlRss&format=rss$token_auth_url&translateColumnNames=1'>10 days</a>";
}
$exampleUrl = $prefixUrls . $exampleUrl;
$str .= " [ Example in
<a target='_blank' href='$exampleUrl&format=xml$token_auth_url'>XML</a>,
<a target='_blank' href='$exampleUrl&format=JSON$token_auth_url'>Json</a>,
<a target='_blank' href='$exampleUrl&format=Tsv$token_auth_url&translateColumnNames=1'>Tsv (Excel)</a>
$lastNUrls
]";
} else {
$str .= " [ No example available ]";
}
$str .= "</span>";
return $str;
}
/**
* Check if Class contains @hide
*
* @param ReflectionClass $rClass instance of ReflectionMethod
* @return bool
*/
public function checkIfClassCommentContainsHideAnnotation(ReflectionClass $rClass)
{
return false !== strstr($rClass->getDocComment(), '@hide');
}
/**
* Check if Class contains @internal
*
* @param ReflectionClass|\ReflectionMethod $rClass instance of ReflectionMethod
* @return bool
*/
private function checkIfCommentContainsInternalAnnotation($rClass)
{
return false !== strstr($rClass->getDocComment(), '@internal');
}
/**
* Check if documentation contains @hide annotation and deletes it
*
* @param $moduleToCheck
* @return mixed
*/
public function checkDocumentation($moduleToCheck)
{
if (strpos($moduleToCheck, '@hide') == true) {
$moduleToCheck = str_replace(strtok(strstr($moduleToCheck, '@hide'), "\n"), "", $moduleToCheck);
}
return $moduleToCheck;
}
/**
* Returns a string containing links to examples on how to call a given method on a given API
* It will export links to XML, CSV, HTML, JSON, PHP, etc.
* It will not export links for methods such as deleteSite or deleteUser
*
* @param string $class the class
* @param string $methodName the method
* @param array $parametersToSet parameters to set
* @return string|bool when not possible
*/
public function getExampleUrl($class, $methodName, $parametersToSet = array())
{
$knowExampleDefaultParametersValues = array(
'access' => 'view',
'userLogin' => 'test',
'passwordMd5ied' => 'passwordExample',
'email' => 'test@example.org',
'languageCode' => 'fr',
'url' => 'https://divezone.net/',
'pageUrl' => 'https://divezone.net/',
'apiModule' => 'UserCountry',
'apiAction' => 'getCountry',
'lastMinutes' => '30',
'abandonedCarts' => '0',
'segmentName' => 'pageTitle',
'ip' => '194.57.91.215',
'idSites' => '1,2',
'idAlert' => '1',
'seconds' => '3600',
// 'segmentName' => 'browserCode',
);
foreach ($parametersToSet as $name => $value) {
$knowExampleDefaultParametersValues[$name] = $value;
}
// no links for these method names
$doNotPrintExampleForTheseMethods = array(
//Sites
'deleteSite',
'addSite',
'updateSite',
'addSiteAliasUrls',
//Users
'deleteUser',
'addUser',
'updateUser',
'setUserAccess',
//Goals
'addGoal',
'updateGoal',
'deleteGoal',
//Marketplace
'deleteLicenseKey'
);
if (in_array($methodName, $doNotPrintExampleForTheseMethods)) {
return false;
}
// we try to give an URL example to call the API
$aParameters = Proxy::getInstance()->getParametersList($class, $methodName);
$aParameters['format'] = false;
$aParameters['hideIdSubDatable'] = false;
$aParameters['serialize'] = false;
$aParameters['language'] = false;
$aParameters['translateColumnNames'] = false;
$aParameters['label'] = false;
$aParameters['labelSeries'] = false;
$aParameters['flat'] = false;
$aParameters['include_aggregate_rows'] = false;
$aParameters['filter_offset'] = false;
$aParameters['filter_limit'] = false;
$aParameters['filter_sort_column'] = false;
$aParameters['filter_sort_order'] = false;
$aParameters['filter_excludelowpop'] = false;
$aParameters['filter_excludelowpop_value'] = false;
$aParameters['filter_column_recursive'] = false;
$aParameters['filter_pattern'] = false;
$aParameters['filter_pattern_recursive'] = false;
$aParameters['filter_truncate'] = false;
$aParameters['hideColumns'] = false;
$aParameters['hideColumnsRecursively'] = false;
$aParameters['showColumns'] = false;
$aParameters['pivotBy'] = false;
$aParameters['pivotByColumn'] = false;
$aParameters['pivotByColumnLimit'] = false;
$aParameters['disable_queued_filters'] = false;
$aParameters['disable_generic_filters'] = false;
$aParameters['expanded'] = false;
$aParameters['idDimenson'] = false;
$aParameters['format_metrics'] = false;
$aParameters['compare'] = false;
$aParameters['compareDates'] = false;
$aParameters['comparePeriods'] = false;
$aParameters['compareSegments'] = false;
$aParameters['comparisonIdSubtables'] = false;
$aParameters['invert_compare_change_compute'] = false;
$entityNames = StaticContainer::get('entities.idNames');
foreach ($entityNames as $entityName) {
if (isset($aParameters[$entityName])) {
continue;
}
$aParameters[$entityName] = false;
}
$moduleName = Proxy::getInstance()->getModuleNameFromClassName($class);
$aParameters = array_merge(array('module' => 'API', 'method' => $moduleName . '.' . $methodName), $aParameters);
foreach ($aParameters as $nameVariable => &$defaultValue) {
if (isset($knowExampleDefaultParametersValues[$nameVariable])) {
$defaultValue = $knowExampleDefaultParametersValues[$nameVariable];
} // if there isn't a default value for a given parameter,
// we need a 'know default value' or we can't generate the link
elseif ($defaultValue instanceof NoDefaultValue) {
return false;
}
}
return '?' . Url::getQueryStringFromParameters($aParameters);
}
/**
* Returns the methods $class.$name parameters (and default value if provided) as a string.
*
* @param string $class The class name
* @param string $name The method name
* @return string For example "(idSite, period, date = 'today')"
*/
protected function getParametersString($class, $name)
{
$aParameters = Proxy::getInstance()->getParametersList($class, $name);
$asParameters = array();
foreach ($aParameters as $nameVariable => $defaultValue) {
// Do not show API parameters starting with _
// They are supposed to be used only in internal API calls
if (strpos($nameVariable, '_') === 0) {
continue;
}
$str = $nameVariable;
if (!($defaultValue instanceof NoDefaultValue)) {
if (is_array($defaultValue)) {
$str .= " = 'Array'";
} else {
$str .= " = '$defaultValue'";
}
}
$asParameters[] = $str;
}
$sParameters = implode(", ", $asParameters);
return "($sParameters)";
}
/**
* @param $outputExampleUrls
* @param $prefixUrls
* @param $displayTitlesAsAngularDirective
* @return array
*/
protected function generateDocumentation($outputExampleUrls, $prefixUrls, $displayTitlesAsAngularDirective)
{
$str = $toc = '';
foreach (Proxy::getInstance()->getMetadata() as $class => $info) {
$moduleName = Proxy::getInstance()->getModuleNameFromClassName($class);
$rClass = new ReflectionClass($class);
if (!Piwik::hasUserSuperUserAccess() && $this->checkIfClassCommentContainsHideAnnotation($rClass)) {
continue;
}
if ($this->checkIfCommentContainsInternalAnnotation($rClass)) {
continue;
}
$toDisplay = $this->prepareModulesAndMethods($info, $moduleName);
foreach ($toDisplay as $moduleName => $methods) {
foreach ($methods as $index => $method) {
if (!method_exists($class, $method)) { // method is handled through API.Request.intercept event
continue;
}
$reflectionMethod = new \ReflectionMethod($class, $method);
if ($this->checkIfCommentContainsInternalAnnotation($reflectionMethod)) {
unset($toDisplay[$moduleName][$index]);
}
}
if (empty($toDisplay[$moduleName])) {
unset($toDisplay[$moduleName]);
}
}
foreach ($toDisplay as $moduleName => $methods) {
$toc .= $this->prepareModuleToDisplay($moduleName);
$str .= $this->prepareMethodToDisplay($moduleName, $info, $methods, $class, $outputExampleUrls, $prefixUrls, $displayTitlesAsAngularDirective);
}
}
return array($toc, $str);
}
}

46
matomo/core/API/Inconsistencies.php Normal file
View file

@ -0,0 +1,46 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\API;
/**
* Contains logic to replicate inconsistencies in Piwik's API. This class exists
* to provide a way to clean up existing Piwik code and behavior without breaking
* backwards compatibility immediately.
*
* Code that handles the case when the 'format_metrics' query parameter value is
* 'bc' should be removed as well. This code is in API\Request and DataTablePostProcessor.
*
* Should be removed before releasing Piwik 3.0.
*/
class Inconsistencies
{
/**
* In Piwik 2.X and below, the "raw" API would format percent values but no others.
* This method returns the list of percent metrics that were returned from the API
* formatted so we can maintain BC.
*
* Used by DataTablePostProcessor.
*/
public function getPercentMetricsToFormat()
{
return array(
'bounce_rate',
'conversion_rate',
'abandoned_rate',
'interaction_rate',
'exit_rate',
'bounce_rate_returning',
'nb_visits_percentage',
'/.*_evolution/',
'/step_.*_rate/',
'/funnel_.*_rate/',
'/form_.*_rate/',
'/field_.*_rate/',
);
}
}

18
matomo/core/API/NoDefaultValue.php Normal file
View file

@ -0,0 +1,18 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
/**
* To differentiate between "no value" and default value of null
*
*/
class NoDefaultValue
{
}

581
matomo/core/API/Proxy.php Normal file
View file

@ -0,0 +1,581 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
use Exception;
use Piwik\Common;
use Piwik\Container\StaticContainer;
use Piwik\Context;
use Piwik\Piwik;
use Piwik\Plugin\Manager;
use ReflectionClass;
use ReflectionMethod;
// prevent upgrade error eg from Matomo 3.x to Matomo 4.x. Refs https://github.com/matomo-org/matomo/pull/16468
// the `false` is important otherwise it would fail and try to load the proxy.php file again.
if (!class_exists('Piwik\API\NoDefaultValue', false)) {
class NoDefaultValue
{
}
}
/**
* Proxy is a singleton that has the knowledge of every method available, their parameters
* and default values.
* Proxy receives all the API calls requests via call() and forwards them to the right
* object, with the parameters in the right order.
*
* It will also log the performance of API calls (time spent, parameter values, etc.) if logger available
*/
class Proxy
{
// array of already registered plugins names
protected $alreadyRegistered = array();
protected $metadataArray = array();
private $hideIgnoredFunctions = true;
// when a parameter doesn't have a default value we use this
private $noDefaultValue;
public function __construct()
{
$this->noDefaultValue = new NoDefaultValue();
}
public static function getInstance()
{
return StaticContainer::get(self::class);
}
/**
* Returns array containing reflection meta data for all the loaded classes
* eg. number of parameters, method names, etc.
*
* @return array
*/
public function getMetadata()
{
ksort($this->metadataArray);
return $this->metadataArray;
}
/**
* Registers the API information of a given module.
*
* The module to be registered must be
* - a singleton (providing a getInstance() method)
* - the API file must be located in plugins/ModuleName/API.php
* for example plugins/Referrers/API.php
*
* The method will introspect the methods, their parameters, etc.
*
* @param string $className ModuleName eg. "API"
*/
public function registerClass($className)
{
if (isset($this->alreadyRegistered[$className])) {
return;
}
$this->includeApiFile($className);
$this->checkClassIsSingleton($className);
$rClass = new ReflectionClass($className);
if (!$this->shouldHideAPIMethod($rClass->getDocComment())) {
foreach ($rClass->getMethods() as $method) {
$this->loadMethodMetadata($className, $method);
}
$this->setDocumentation($rClass, $className);
$this->alreadyRegistered[$className] = true;
}
}
/**
* Will be displayed in the API page
*
* @param ReflectionClass $rClass Instance of ReflectionClass
* @param string $className Name of the class
*/
private function setDocumentation($rClass, $className)
{
// Doc comment
$doc = $rClass->getDocComment();
$doc = str_replace(" * " . PHP_EOL, "<br>", $doc);
// boldify the first line only if there is more than one line, otherwise too much bold
if (substr_count($doc, '<br>') > 1) {
$firstLineBreak = strpos($doc, "<br>");
$doc = "<div class='apiFirstLine'>" . substr($doc, 0, $firstLineBreak) . "</div>" . substr($doc, $firstLineBreak + strlen("<br>"));
}
$doc = preg_replace("/(@package)[a-z _A-Z]*/", "", $doc);
$doc = preg_replace("/(@method).*/", "", $doc);
$doc = str_replace(array("\t", "\n", "/**", "*/", " * ", " *", " ", "\t*", " * @package"), " ", $doc);
// replace 'foo' and `bar` and "foobar" with code blocks... much magic
$doc = preg_replace('/`(.*?)`/', '<code>$1</code>', $doc);
$this->metadataArray[$className]['__documentation'] = $doc;
}
/**
* Returns number of classes already loaded
* @return int
*/
public function getCountRegisteredClasses()
{
return count($this->alreadyRegistered);
}
/**
* Will execute $className->$methodName($parametersValues)
* If any error is detected (wrong number of parameters, method not found, class not found, etc.)
* it will throw an exception
*
* It also logs the API calls, with the parameters values, the returned value, the performance, etc.
* You can enable logging in config/global.ini.php (log_api_call)
*
* @param string $className The class name (eg. API)
* @param string $methodName The method name
* @param array $parametersRequest The parameters pairs (name=>value)
*
* @return mixed|null
* @throws Exception|\Piwik\NoAccessException
*/
public function call($className, $methodName, $parametersRequest)
{
// Temporarily sets the Request array to this API call context
return Context::executeWithQueryParameters($parametersRequest, function () use ($className, $methodName, $parametersRequest) {
$returnedValue = null;
$this->registerClass($className);
// instantiate the object
$object = $className::getInstance();
// check method exists
$this->checkMethodExists($className, $methodName);
// get the list of parameters required by the method
$parameterNamesDefaultValues = $this->getParametersList($className, $methodName);
// load parameters in the right order, etc.
$finalParameters = $this->getRequestParametersArray($parameterNamesDefaultValues, $parametersRequest);
// allow plugins to manipulate the value
$pluginName = $this->getModuleNameFromClassName($className);
$returnedValue = null;
/**
* Triggered before an API request is dispatched.
*
* This event can be used to modify the arguments passed to one or more API methods.
*
* **Example**
*
* Piwik::addAction('API.Request.dispatch', function (&$parameters, $pluginName, $methodName) {
* if ($pluginName == 'Actions') {
* if ($methodName == 'getPageUrls') {
* // ... do something ...
* } else {
* // ... do something else ...
* }
* }
* });
*
* @param array &$finalParameters List of parameters that will be passed to the API method.
* @param string $pluginName The name of the plugin the API method belongs to.
* @param string $methodName The name of the API method that will be called.
*/
Piwik::postEvent('API.Request.dispatch', array(&$finalParameters, $pluginName, $methodName));
/**
* Triggered before an API request is dispatched.
*
* This event exists for convenience and is triggered directly after the {@hook API.Request.dispatch}
* event is triggered. It can be used to modify the arguments passed to a **single** API method.
*
* _Note: This is can be accomplished with the {@hook API.Request.dispatch} event as well, however
* event handlers for that event will have to do more work._
*
* **Example**
*
* Piwik::addAction('API.Actions.getPageUrls', function (&$parameters) {
* // force use of a single website. for some reason.
* $parameters['idSite'] = 1;
* });
*
* @param array &$finalParameters List of parameters that will be passed to the API method.
*/
Piwik::postEvent(sprintf('API.%s.%s', $pluginName, $methodName), array(&$finalParameters));
/**
* Triggered before an API request is dispatched.
*
* Use this event to intercept an API request and execute your own code instead. If you set
* `$returnedValue` in a handler for this event, the original API method will not be executed,
* and the result will be what you set in the event handler.
*
* @param mixed &$returnedValue Set this to set the result and preempt normal API invocation.
* @param array &$finalParameters List of parameters that will be passed to the API method.
* @param string $pluginName The name of the plugin the API method belongs to.
* @param string $methodName The name of the API method that will be called.
* @param array $parametersRequest The query parameters for this request.
*/
Piwik::postEvent('API.Request.intercept', [&$returnedValue, $finalParameters, $pluginName, $methodName, $parametersRequest]);
$apiParametersInCorrectOrder = array();
foreach ($parameterNamesDefaultValues as $name => $defaultValue) {
if (isset($finalParameters[$name]) || array_key_exists($name, $finalParameters)) {
$apiParametersInCorrectOrder[] = $finalParameters[$name];
}
}
// call the method if a hook hasn't already set an output variable
if ($returnedValue === null) {
$returnedValue = call_user_func_array(array($object, $methodName), $apiParametersInCorrectOrder);
}
$endHookParams = array(
&$returnedValue,
array('className' => $className,
'module' => $pluginName,
'action' => $methodName,
'parameters' => $finalParameters)
);
/**
* Triggered directly after an API request is dispatched.
*
* This event exists for convenience and is triggered immediately before the
* {@hook API.Request.dispatch.end} event. It can be used to modify the output of a **single**
* API method.
*
* _Note: This can be accomplished with the {@hook API.Request.dispatch.end} event as well,
* however event handlers for that event will have to do more work._
*
* **Example**
*
* // append (0 hits) to the end of row labels whose row has 0 hits
* Piwik::addAction('API.Actions.getPageUrls', function (&$returnValue, $info)) {
* $returnValue->filter('ColumnCallbackReplace', 'label', function ($label, $hits) {
* if ($hits === 0) {
* return $label . " (0 hits)";
* } else {
* return $label;
* }
* }, null, array('nb_hits'));
* }
*
* @param mixed &$returnedValue The API method's return value. Can be an object, such as a
* {@link Piwik\DataTable DataTable} instance.
* could be a {@link Piwik\DataTable DataTable}.
* @param array $extraInfo An array holding information regarding the API request. Will
* contain the following data:
*
* - **className**: The namespace-d class name of the API instance
* that's being called.
* - **module**: The name of the plugin the API request was
* dispatched to.
* - **action**: The name of the API method that was executed.
* - **parameters**: The array of parameters passed to the API
* method.
*/
Piwik::postEvent(sprintf('API.%s.%s.end', $pluginName, $methodName), $endHookParams);
/**
* Triggered directly after an API request is dispatched.
*
* This event can be used to modify the output of any API method.
*
* **Example**
*
* // append (0 hits) to the end of row labels whose row has 0 hits for any report that has the 'nb_hits' metric
* Piwik::addAction('API.Actions.getPageUrls.end', function (&$returnValue, $info)) {
* // don't process non-DataTable reports and reports that don't have the nb_hits column
* if (!($returnValue instanceof DataTableInterface)
* || in_array('nb_hits', $returnValue->getColumns())
* ) {
* return;
* }
*
* $returnValue->filter('ColumnCallbackReplace', 'label', function ($label, $hits) {
* if ($hits === 0) {
* return $label . " (0 hits)";
* } else {
* return $label;
* }
* }, null, array('nb_hits'));
* }
*
* @param mixed &$returnedValue The API method's return value. Can be an object, such as a
* {@link Piwik\DataTable DataTable} instance.
* @param array $extraInfo An array holding information regarding the API request. Will
* contain the following data:
*
* - **className**: The namespace-d class name of the API instance
* that's being called.
* - **module**: The name of the plugin the API request was
* dispatched to.
* - **action**: The name of the API method that was executed.
* - **parameters**: The array of parameters passed to the API
* method.
*/
Piwik::postEvent('API.Request.dispatch.end', $endHookParams);
return $returnedValue;
});
}
/**
* Returns the parameters names and default values for the method $name
* of the class $class
*
* @param string $class The class name
* @param string $name The method name
* @return array Format array(
* 'testParameter' => null, // no default value
* 'life' => 42, // default value = 42
* 'date' => 'yesterday',
* );
*/
public function getParametersList($class, $name)
{
return $this->metadataArray[$class][$name]['parameters'];
}
/**
* Check if given method name is deprecated or not.
*/
public function isDeprecatedMethod($class, $methodName)
{
return $this->metadataArray[$class][$methodName]['isDeprecated'];
}
/**
* Returns the 'moduleName' part of '\\Piwik\\Plugins\\moduleName\\API'
*
* @param string $className "API"
* @return string "Referrers"
*/
public function getModuleNameFromClassName($className)
{
return str_replace(array('\\Piwik\\Plugins\\', '\\API'), '', $className);
}
public function isExistingApiAction($pluginName, $apiAction)
{
$namespacedApiClassName = "\\Piwik\\Plugins\\$pluginName\\API";
$api = $namespacedApiClassName::getInstance();
return method_exists($api, $apiAction);
}
public function buildApiActionName($pluginName, $apiAction)
{
return sprintf("%s.%s", $pluginName, $apiAction);
}
/**
* Sets whether to hide '@ignore'd functions from method metadata or not.
*
* @param bool $hideIgnoredFunctions
*/
public function setHideIgnoredFunctions($hideIgnoredFunctions)
{
$this->hideIgnoredFunctions = $hideIgnoredFunctions;
// make sure metadata gets reloaded
$this->alreadyRegistered = array();
$this->metadataArray = array();
}
/**
* Returns an array containing the values of the parameters to pass to the method to call
*
* @param array $requiredParameters array of (parameter name, default value)
* @param array $parametersRequest
* @throws Exception
* @return array values to pass to the function call
*/
private function getRequestParametersArray($requiredParameters, $parametersRequest)
{
$finalParameters = array();
foreach ($requiredParameters as $name => $defaultValue) {
try {
if ($defaultValue instanceof NoDefaultValue) {
$requestValue = Common::getRequestVar($name, null, null, $parametersRequest);
} else {
try {
if ($name == 'segment' && !empty($parametersRequest['segment'])) {
// segment parameter is an exception: we do not want to sanitize user input or it would break the segment encoding
$requestValue = ($parametersRequest['segment']);
} else {
$requestValue = Common::getRequestVar($name, $defaultValue, null, $parametersRequest);
}
} catch (Exception $e) {
// Special case: empty parameter in the URL, should return the empty string
if (isset($parametersRequest[$name])
&& $parametersRequest[$name] === ''
) {
$requestValue = '';
} else {
$requestValue = $defaultValue;
}
}
}
} catch (Exception $e) {
throw new Exception(Piwik::translate('General_PleaseSpecifyValue', array($name)));
}
$finalParameters[$name] = $requestValue;
}
return $finalParameters;
}
/**
* Includes the class API by looking up plugins/xxx/API.php
*
* @param string $fileName api class name eg. "API"
* @throws Exception
*/
private function includeApiFile($fileName)
{
$module = self::getModuleNameFromClassName($fileName);
$path = Manager::getPluginDirectory($module) . '/API.php';
if (is_readable($path)) {
require_once $path; // prefixed by PIWIK_INCLUDE_PATH
} else {
throw new Exception("API module $module not found.");
}
}
/**
* @param string $class name of a class
* @param ReflectionMethod $method instance of ReflectionMethod
*/
private function loadMethodMetadata($class, $method)
{
if (!$this->checkIfMethodIsAvailable($method)) {
return;
}
$name = $method->getName();
$parameters = $method->getParameters();
$docComment = $method->getDocComment();
$aParameters = array();
foreach ($parameters as $parameter) {
$nameVariable = $parameter->getName();
$defaultValue = $this->noDefaultValue;
if ($parameter->isDefaultValueAvailable()) {
$defaultValue = $parameter->getDefaultValue();
}
$aParameters[$nameVariable] = $defaultValue;
}
$this->metadataArray[$class][$name]['parameters'] = $aParameters;
$this->metadataArray[$class][$name]['numberOfRequiredParameters'] = $method->getNumberOfRequiredParameters();
$this->metadataArray[$class][$name]['isDeprecated'] = false !== strstr($docComment, '@deprecated');
}
/**
* Checks that the method exists in the class
*
* @param string $className The class name
* @param string $methodName The method name
* @throws Exception If the method is not found
*/
private function checkMethodExists($className, $methodName)
{
if (!$this->isMethodAvailable($className, $methodName)) {
throw new Exception(Piwik::translate('General_ExceptionMethodNotFound', array($methodName, $className)));
}
}
/**
* @param $docComment
* @return bool
*/
public function shouldHideAPIMethod($docComment)
{
$hideLine = strstr($docComment, '@hide');
if ($hideLine === false) {
return false;
}
$hideLine = trim($hideLine);
$hideLine .= ' ';
$token = trim(strtok($hideLine, " "), "\n");
$hide = false;
if (!empty($token)) {
/**
* This event exists for checking whether a Plugin API class or a Plugin API method tagged
* with a `@hideXYZ` should be hidden in the API listing.
*
* @param bool &$hide whether to hide APIs tagged with $token should be displayed.
*/
Piwik::postEvent(sprintf('API.DocumentationGenerator.%s', $token), array(&$hide));
}
return $hide;
}
/**
* @param ReflectionMethod $method
* @return bool
*/
protected function checkIfMethodIsAvailable(ReflectionMethod $method)
{
if (!$method->isPublic() || $method->isConstructor() || $method->getName() === 'getInstance') {
return false;
}
if ($this->hideIgnoredFunctions && false !== strstr($method->getDocComment(), '@ignore')) {
return false;
}
if ($this->shouldHideAPIMethod($method->getDocComment())) {
return false;
}
return true;
}
/**
* Returns true if the method is found in the API of the given class name.
*
* @param string $className The class name
* @param string $methodName The method name
* @return bool
*/
private function isMethodAvailable($className, $methodName)
{
return isset($this->metadataArray[$className][$methodName]);
}
/**
* Checks that the class is a Singleton (presence of the getInstance() method)
*
* @param string $className The class name
* @throws Exception If the class is not a Singleton
*/
private function checkClassIsSingleton($className)
{
if (!method_exists($className, "getInstance")) {
throw new Exception("$className that provide an API must be Singleton and have a 'public static function getInstance()' method.");
}
}
}

705
matomo/core/API/Request.php Normal file
View file

@ -0,0 +1,705 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
use Exception;
use Piwik\Access;
use Piwik\Cache;
use Piwik\Common;
use Piwik\Config;
use Piwik\Container\StaticContainer;
use Piwik\Context;
use Piwik\DataTable;
use Piwik\Exception\PluginDeactivatedException;
use Piwik\IP;
use Piwik\Piwik;
use Piwik\Plugin\Manager as PluginManager;
use Piwik\Plugins\CoreHome\LoginAllowlist;
use Piwik\SettingsServer;
use Piwik\Url;
use Piwik\UrlHelper;
use Psr\Log\LoggerInterface;
/**
* Dispatches API requests to the appropriate API method.
*
* The Request class is used throughout Piwik to call API methods. The difference
* between using Request and calling API methods directly is that Request
* will do more after calling the API including: applying generic filters, applying queued filters,
* and handling the **flat** and **label** query parameters.
*
* Additionally, the Request class will **forward current query parameters** to the request
* which is more convenient than calling {@link Piwik\Common::getRequestVar()} many times over.
*
* In most cases, using a Request object to query the API is the correct approach.
*
* ### Post-processing
*
* The return value of API methods undergo some extra processing before being returned by Request.
*
* ### Output Formats
*
* The value returned by Request will be serialized to a certain format before being returned.
*
* ### Examples
*
* **Basic Usage**
*
* $request = new Request('method=UserLanguage.getLanguage&idSite=1&date=yesterday&period=week'
* . '&format=xml&filter_limit=5&filter_offset=0')
* $result = $request->process();
* echo $result;
*
* **Getting a unrendered DataTable**
*
* // use the convenience method 'processRequest'
* $dataTable = Request::processRequest('UserLanguage.getLanguage', array(
* 'idSite' => 1,
* 'date' => 'yesterday',
* 'period' => 'week',
* 'filter_limit' => 5,
* 'filter_offset' => 0
*
* 'format' => 'original', // this is the important bit
* ));
* echo "This DataTable has " . $dataTable->getRowsCount() . " rows.";
*
* @see http://piwik.org/docs/analytics-api
* @api
*/
class Request
{
/**
* The count of nested API request invocations. Used to determine if the currently executing request is the root or not.
*
* @var int
*/
private static $nestedApiInvocationCount = 0;
private $request = null;
/**
* Converts the supplied request string into an array of query parameter name/value
* mappings. The current query parameters (everything in `$_GET` and `$_POST`) are
* forwarded to request array before it is returned.
*
* @param string|array|null $request The base request string or array, eg,
* `'module=UserLanguage&action=getLanguage'`.
* @param array $defaultRequest Default query parameters. If a query parameter is absent in `$request`, it will be loaded
* from this. Defaults to `$_GET + $_POST`.
* @return array
*/
public static function getRequestArrayFromString($request, $defaultRequest = null)
{
if ($defaultRequest === null) {
$defaultRequest = self::getDefaultRequest();
$requestRaw = self::getRequestParametersGET();
if (!empty($requestRaw['segment'])) {
$defaultRequest['segment'] = $requestRaw['segment'];
}
// Only default to formatting metrics if the request doesn't already contain the format metrics parameter
if (!isset($defaultRequest['format_metrics']) && !isset($request['format_metrics'])) {
$defaultRequest['format_metrics'] = 'bc';
}
}
$requestArray = $defaultRequest;
if (!is_null($request)) {
if (is_array($request)) {
$requestParsed = $request;
} else {
$request = trim($request);
$request = str_replace(array("\n", "\t"), '', $request);
$requestParsed = UrlHelper::getArrayFromQueryString($request);
}
$requestArray = $requestParsed + $defaultRequest;
}
foreach ($requestArray as &$element) {
if (!is_array($element)) {
$element = trim((string) $element);
}
}
return $requestArray;
}
/**
* Constructor.
*
* @param string|array $request Query string that defines the API call (must at least contain a **method** parameter),
* eg, `'method=UserLanguage.getLanguage&idSite=1&date=yesterday&period=week&format=xml'`
* If a request is not provided, then we use the values in the `$_GET` and `$_POST`
* superglobals.
* @param array $defaultRequest Default query parameters. If a query parameter is absent in `$request`, it will be loaded
* from this. Defaults to `$_GET + $_POST`.
*/
public function __construct($request = null, $defaultRequest = null)
{
$this->request = self::getRequestArrayFromString($request, $defaultRequest);
$this->sanitizeRequest();
$this->renameModuleAndActionInRequest();
}
/**
* For backward compatibility: Piwik API still works if module=Referers,
* we rewrite to correct renamed plugin: Referrers
*
* @param $module
* @param $action
* @return array( $module, $action )
* @ignore
*/
public static function getRenamedModuleAndAction($module, $action)
{
/**
* This event is posted in the Request dispatcher and can be used
* to overwrite the Module and Action to dispatch.
* This is useful when some Controller methods or API methods have been renamed or moved to another plugin.
*
* @param $module string
* @param $action string
*/
Piwik::postEvent('Request.getRenamedModuleAndAction', array(&$module, &$action));
return array($module, $action);
}
/**
* Make sure that the request contains no logical errors
*/
private function sanitizeRequest()
{
// The label filter does not work with expanded=1 because the data table IDs have a different meaning
// depending on whether the table has been loaded yet. expanded=1 causes all tables to be loaded, which
// is why the label filter can't descend when a recursive label has been requested.
// To fix this, we remove the expanded parameter if a label parameter is set.
if (isset($this->request['label']) && !empty($this->request['label'])
&& isset($this->request['expanded']) && $this->request['expanded']
) {
unset($this->request['expanded']);
}
}
/**
* Dispatches the API request to the appropriate API method and returns the result
* after post-processing.
*
* Post-processing includes:
*
* - flattening if **flat** is 0
* - running generic filters unless **disable_generic_filters** is set to 1
* - URL decoding label column values
* - running queued filters unless **disable_queued_filters** is set to 1
* - removing columns based on the values of the **hideColumns** and **showColumns** query parameters
* - filtering rows if the **label** query parameter is set
* - converting the result to the appropriate format (ie, XML, JSON, etc.)
*
* If `'original'` is supplied for the output format, the result is returned as a PHP
* object.
*
* @throws PluginDeactivatedException if the module plugin is not activated.
* @throws Exception if the requested API method cannot be called, if required parameters for the
* API method are missing or if the API method throws an exception and the **format**
* query parameter is **original**.
* @return DataTable|Map|string The data resulting from the API call.
*/
public function process()
{
$shouldReloadAuth = false;
try {
++self::$nestedApiInvocationCount;
// read the format requested for the output data
$outputFormat = strtolower(Common::getRequestVar('format', 'xml', 'string', $this->request));
$disablePostProcessing = $this->shouldDisablePostProcessing();
// create the response
$response = new ResponseBuilder($outputFormat, $this->request);
if ($disablePostProcessing) {
$response->disableDataTablePostProcessor();
}
$corsHandler = new CORSHandler();
$corsHandler->handle();
$tokenAuth = Common::getRequestVar('token_auth', '', 'string', $this->request);
// IP check is needed here as we cannot listen to API.Request.authenticate as it would then not return proper API format response.
// We can also not do it by listening to API.Request.dispatch as by then the user is already authenticated and we want to make sure
// to not expose any information in case the IP is not allowed.
$list = new LoginAllowlist();
if ($list->shouldCheckAllowlist() && $list->shouldAllowlistApplyToAPI()) {
$ip = IP::getIpFromHeader();
$list->checkIsAllowed($ip);
}
// read parameters
$moduleMethod = Common::getRequestVar('method', null, 'string', $this->request);
list($module, $method) = $this->extractModuleAndMethod($moduleMethod);
list($module, $method) = self::getRenamedModuleAndAction($module, $method);
PluginManager::getInstance()->checkIsPluginActivated($module);
$apiClassName = self::getClassNameAPI($module);
if ($shouldReloadAuth = self::shouldReloadAuthUsingTokenAuth($this->request)) {
$access = Access::getInstance();
$tokenAuthToRestore = $access->getTokenAuth();
$hadSuperUserAccess = $access->hasSuperUserAccess();
self::forceReloadAuthUsingTokenAuth($tokenAuth);
}
// call the method
$returnedValue = Proxy::getInstance()->call($apiClassName, $method, $this->request);
// get the response with the request query parameters loaded, since DataTablePost processor will use the Report
// class instance, which may inspect the query parameters. (eg, it may look for the idCustomReport parameters
// which may only exist in $this->request, if the request was called programmatically)
$toReturn = Context::executeWithQueryParameters($this->request, function () use ($response, $returnedValue, $module, $method) {
return $response->getResponse($returnedValue, $module, $method);
});
} catch (Exception $e) {
StaticContainer::get(LoggerInterface::class)->error('Uncaught exception in API: {exception}', [
'exception' => $e,
'ignoreInScreenWriter' => true,
]);
if (empty($response)) {
$response = new ResponseBuilder('console', $this->request);
}
$toReturn = $response->getResponseException($e);
} finally {
--self::$nestedApiInvocationCount;
}
if ($shouldReloadAuth) {
$this->restoreAuthUsingTokenAuth($tokenAuthToRestore, $hadSuperUserAccess);
}
return $toReturn;
}
private function restoreAuthUsingTokenAuth($tokenToRestore, $hadSuperUserAccess)
{
// if we would not make sure to unset super user access, the tokenAuth would be not authenticated and any
// token would just keep super user access (eg if the token that was reloaded before had super user access)
Access::getInstance()->setSuperUserAccess(false);
// we need to restore by reloading the tokenAuth as some permissions could have been removed in the API
// request etc. Otherwise we could just store a clone of Access::getInstance() and restore here
self::forceReloadAuthUsingTokenAuth($tokenToRestore);
if ($hadSuperUserAccess && !Access::getInstance()->hasSuperUserAccess()) {
// we are in context of `doAsSuperUser()` and need to restore this behaviour
Access::getInstance()->setSuperUserAccess(true);
}
}
/**
* Returns the name of a plugin's API class by plugin name.
*
* @param string $plugin The plugin name, eg, `'Referrers'`.
* @return string The fully qualified API class name, eg, `'\Piwik\Plugins\Referrers\API'`.
*/
public static function getClassNameAPI($plugin)
{
return sprintf('\Piwik\Plugins\%s\API', $plugin);
}
/**
* @ignore
* @internal
* @param string $currentApiMethod
*/
public static function setIsRootRequestApiRequest($currentApiMethod)
{
Cache::getTransientCache()->save('API.setIsRootRequestApiRequest', $currentApiMethod);
}
/**
* @ignore
* @internal
* @return string current Api Method if it is an api request
*/
public static function getRootApiRequestMethod()
{
return Cache::getTransientCache()->fetch('API.setIsRootRequestApiRequest');
}
/**
* Detect if the root request (the actual request) is an API request or not. To detect whether an API is currently
* request within any request, have a look at {@link isApiRequest()}.
*
* @return bool
* @throws Exception
*/
public static function isRootRequestApiRequest()
{
$apiMethod = Cache::getTransientCache()->fetch('API.setIsRootRequestApiRequest');
return !empty($apiMethod);
}
/**
* Checks if the currently executing API request is the root API request or not.
*
* Note: the "root" API request is the first request made. Within that request, further API methods
* can be called programmatically. These requests are considered "child" API requests.
*
* @return bool
* @throws Exception
*/
public static function isCurrentApiRequestTheRootApiRequest()
{
return self::$nestedApiInvocationCount == 1;
}
/**
* Detect if request is an API request. Meaning the module is 'API' and an API method having a valid format was
* specified. Note that this method will return true even if the actual request is for example a regular UI
* reporting page request but within this request we are currently processing an API request (eg a
* controller calls Request::processRequest('API.getMatomoVersion')). To find out if the root request is an API
* request or not, call {@link isRootRequestApiRequest()}
*
* @param array $request eg array('module' => 'API', 'method' => 'Test.getMethod')
* @return bool
* @throws Exception
*/
public static function isApiRequest($request)
{
$method = self::getMethodIfApiRequest($request);
return !empty($method);
}
/**
* Returns the current API method being executed, if the current request is an API request.
*
* @param array $request eg array('module' => 'API', 'method' => 'Test.getMethod')
* @return string|null
* @throws Exception
*/
public static function getMethodIfApiRequest($request)
{
$module = Common::getRequestVar('module', '', 'string', $request);
$method = Common::getRequestVar('method', '', 'string', $request);
$isApi = $module === 'API' && !empty($method) && (count(explode('.', $method)) === 2);
return $isApi ? $method : null;
}
/**
* If the token_auth is found in the $request parameter,
* the current session will be authenticated using this token_auth.
* It will overwrite the previous Auth object.
*
* @param array $request If null, uses the default request ($_GET)
* @return void
* @ignore
*/
public static function reloadAuthUsingTokenAuth($request = null)
{
// if a token_auth is specified in the API request, we load the right permissions
$token_auth = Common::getRequestVar('token_auth', '', 'string', $request);
if (self::shouldReloadAuthUsingTokenAuth($request)) {
self::forceReloadAuthUsingTokenAuth($token_auth);
}
}
/**
* The current session will be authenticated using this token_auth.
* It will overwrite the previous Auth object.
*
* @param string $tokenAuth
* @return void
*/
private static function forceReloadAuthUsingTokenAuth($tokenAuth)
{
/**
* Triggered when authenticating an API request, but only if the **token_auth**
* query parameter is found in the request.
*
* Plugins that provide authentication capabilities should subscribe to this event
* and make sure the global authentication object (the object returned by `StaticContainer::get('Piwik\Auth')`)
* is setup to use `$token_auth` when its `authenticate()` method is executed.
*
* @param string $token_auth The value of the **token_auth** query parameter.
*/
Piwik::postEvent('API.Request.authenticate', array($tokenAuth));
if (!Access::getInstance()->reloadAccess() && $tokenAuth && $tokenAuth !== 'anonymous') {
/**
* @ignore
* @internal
*/
Piwik::postEvent('API.Request.authenticate.failed');
}
SettingsServer::raiseMemoryLimitIfNecessary();
}
/**
* Needs to be called AFTER the user has been authenticated using a token.
*
* @internal
* @ignore
* @param string $module
* @param string $action
* @throws Exception
*/
public static function checkTokenAuthIsNotLimited($module, $action)
{
$isApi = ($module === 'API' && (empty($action) || $action === 'index'));
if ($isApi
|| Common::isPhpCliMode()
) {
return;
}
if (Access::getInstance()->hasSuperUserAccess()) {
$ex = new \Piwik\Exception\Exception(Piwik::translate('Widgetize_TooHighAccessLevel', ['<a href="https://matomo.org/faq/troubleshooting/faq_147/" rel="noreferrer noopener">', '</a>']));
$ex->setIsHtmlMessage();
throw $ex;
}
$allowWriteAmin = Config::getInstance()->General['enable_framed_allow_write_admin_token_auth'] == 1;
if (Piwik::isUserHasSomeWriteAccess()
&& !$allowWriteAmin
) {
// we allow UI authentication/ embedding widgets / reports etc only for users that have only view
// access. it's mostly there to get users to use auth tokens of view users when embedding reports
// token_auth is fine for API calls since they would be always authenticated later anyway
// token_auth is also fine in CLI mode as eg doAsSuperUser might be used etc
//
// NOTE: this does not apply if the [General] enable_framed_allow_write_admin_token_auth INI
// option is set.
throw new \Exception(Piwik::translate('Widgetize_ViewAccessRequired', ['https://matomo.org/faq/troubleshooting/faq_147/']));
}
}
/**
* @internal
* @ignore
* @param $request
* @return bool
* @throws Exception
*/
public static function shouldReloadAuthUsingTokenAuth($request)
{
if (is_null($request)) {
$request = self::getDefaultRequest();
}
if (!isset($request['token_auth'])) {
// no token is given so we just keep the current loaded user
return false;
}
// a token is specified, we need to reload auth in case it is different than the current one, even if it is empty
$tokenAuth = Common::getRequestVar('token_auth', '', 'string', $request);
// not using !== is on purpose as getTokenAuth() might return null whereas $tokenAuth is '' . In this case
// we do not need to reload.
return $tokenAuth != Access::getInstance()->getTokenAuth();
}
/**
* Returns array($class, $method) from the given string $class.$method
*
* @param string $parameter
* @throws Exception
* @return array
*/
private function extractModuleAndMethod($parameter)
{
$a = explode('.', $parameter);
if (count($a) != 2) {
throw new Exception("The method name is invalid. Expected 'module.methodName'");
}
return $a;
}
/**
* Helper method that processes an API request in one line using the variables in `$_GET`
* and `$_POST`.
*
* @param string $method The API method to call, ie, `'Actions.getPageTitles'`.
* @param array $paramOverride The parameter name-value pairs to use instead of what's
* in `$_GET` & `$_POST`.
* @param array $defaultRequest Default query parameters. If a query parameter is absent in `$request`, it will be loaded
* from this. Defaults to `$_GET + $_POST`.
*
* To avoid using any parameters from $_GET or $_POST, set this to an empty `array()`.
* @return mixed The result of the API request. See {@link process()}.
*/
public static function processRequest($method, $paramOverride = array(), $defaultRequest = null)
{
$params = array();
$params['format'] = 'original';
$params['serialize'] = '0';
$params['module'] = 'API';
$params['method'] = $method;
$params['compare'] = '0';
$params = $paramOverride + $params;
// process request
$request = new Request($params, $defaultRequest);
return $request->process();
}
/**
* Returns the original request parameters in the current query string as an array mapping
* query parameter names with values. The result of this function will not be affected
* by any modifications to `$_GET` and will not include parameters in `$_POST`.
*
* @return array
*/
public static function getRequestParametersGET()
{
if (empty($_SERVER['QUERY_STRING'])) {
return array();
}
$GET = UrlHelper::getArrayFromQueryString($_SERVER['QUERY_STRING']);
return $GET;
}
/**
* Returns the URL for the current requested report w/o any filter parameters.
*
* @param string $module The API module.
* @param string $action The API action.
* @param array $queryParams Query parameter overrides.
* @return string
*/
public static function getBaseReportUrl($module, $action, $queryParams = array())
{
$params = array_merge($queryParams, array('module' => $module, 'action' => $action));
return Request::getCurrentUrlWithoutGenericFilters($params);
}
/**
* Returns the current URL without generic filter query parameters.
*
* @param array $params Query parameter values to override in the new URL.
* @return string
*/
public static function getCurrentUrlWithoutGenericFilters($params)
{
// unset all filter query params so the related report will show up in its default state,
// unless the filter param was in $queryParams
$genericFiltersInfo = DataTableGenericFilter::getGenericFiltersInformation();
foreach ($genericFiltersInfo as $filter) {
foreach ($filter[1] as $queryParamName => $queryParamInfo) {
if (!isset($params[$queryParamName])) {
$params[$queryParamName] = null;
}
}
}
$params['compareDates'] = null;
$params['comparePeriods'] = null;
$params['compareSegments'] = null;
return Url::getCurrentQueryStringWithParametersModified($params);
}
/**
* Returns whether the DataTable result will have to be expanded for the
* current request before rendering.
*
* @return bool
* @ignore
*/
public static function shouldLoadExpanded()
{
// if filter_column_recursive & filter_pattern_recursive are supplied, and flat isn't supplied
// we have to load all the child subtables.
return Common::getRequestVar('filter_column_recursive', false) !== false
&& Common::getRequestVar('filter_pattern_recursive', false) !== false
&& !self::shouldLoadFlatten();
}
/**
* @return bool
*/
public static function shouldLoadFlatten()
{
return Common::getRequestVar('flat', false) == 1;
}
/**
* Returns the segment query parameter from the original request, without modifications.
*
* @return array|bool
*/
public static function getRawSegmentFromRequest()
{
// we need the URL encoded segment parameter, we fetch it from _SERVER['QUERY_STRING'] instead of default URL decoded _GET
$segmentRaw = false;
$segment = Common::getRequestVar('segment', '', 'string');
if (!empty($segment)) {
$request = Request::getRequestParametersGET();
if (!empty($request['segment'])) {
$segmentRaw = $request['segment'];
}
}
return $segmentRaw;
}
private function renameModuleAndActionInRequest()
{
if (empty($this->request['apiModule'])) {
return;
}
if (empty($this->request['apiAction'])) {
$this->request['apiAction'] = null;
}
list($this->request['apiModule'], $this->request['apiAction']) = $this->getRenamedModuleAndAction($this->request['apiModule'], $this->request['apiAction']);
}
/**
* @return array
*/
private static function getDefaultRequest()
{
return $_GET + $_POST;
}
private function shouldDisablePostProcessing()
{
$shouldDisable = false;
/**
* After an API method returns a value, the value is post processed (eg, rows are sorted
* based on the `filter_sort_column` query parameter, rows are truncated based on the
* `filter_limit`/`filter_offset` parameters, amongst other things).
*
* If you're creating a plugin that needs to disable post processing entirely for
* certain requests, use this event.
*
* @param bool &$shouldDisable Set this to true to disable datatable post processing for a request.
* @param array $request The request parameters.
*/
Piwik::postEvent('Request.shouldDisablePostProcessing', [&$shouldDisable, $this->request]);
if (!$shouldDisable) {
$shouldDisable = self::isCurrentApiRequestTheRootApiRequest() &&
Common::getRequestVar('disable_root_datatable_post_processor', 0, 'int', $this->request) == 1;
}
return $shouldDisable;
}
}

245
matomo/core/API/ResponseBuilder.php Normal file
View file

@ -0,0 +1,245 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\API;
use Exception;
use Piwik\Common;
use Piwik\DataTable;
use Piwik\DataTable\Renderer;
use Piwik\DataTable\DataTableInterface;
use Piwik\DataTable\Filter\ColumnDelete;
use Piwik\DataTable\Filter\Pattern;
use Piwik\Http\HttpCodeException;
use Piwik\Plugins\Monolog\Processor\ExceptionToTextProcessor;
/**
*/
class ResponseBuilder
{
private $outputFormat = null;
private $apiRenderer = null;
private $request = null;
private $sendHeader = true;
private $postProcessDataTable = true;
private $apiModule = false;
private $apiMethod = false;
private $shouldPrintBacktrace = false;
/**
* @param string $outputFormat
* @param array $request
*/
public function __construct($outputFormat, $request = array(), $shouldPrintBacktrace = null)
{
$this->outputFormat = $outputFormat;
$this->request = $request;
$this->apiRenderer = ApiRenderer::factory($outputFormat, $request);
$this->shouldPrintBacktrace = $shouldPrintBacktrace === null ? \Piwik_ShouldPrintBackTraceWithMessage() : $shouldPrintBacktrace;
}
public function disableSendHeader()
{
$this->sendHeader = false;
}
public function disableDataTablePostProcessor()
{
$this->postProcessDataTable = false;
}
/**
* This method processes the data resulting from the API call.
*
* - If the data resulted from the API call is a DataTable then
* - we apply the standard filters if the parameters have been found
* in the URL. For example to offset,limit the Table you can add the following parameters to any API
* call that returns a DataTable: filter_limit=10&filter_offset=20
* - we apply the filters that have been previously queued on the DataTable
* @see DataTable::queueFilter()
* - we apply the renderer that generate the DataTable in a given format (XML, PHP, HTML, JSON, etc.)
* the format can be changed using the 'format' parameter in the request.
* Example: format=xml
*
* - If there is nothing returned (void) we display a standard success message
*
* - If there is a PHP array returned, we try to convert it to a dataTable
* It is then possible to convert this datatable to any requested format (xml/etc)
*
* - If a bool is returned we convert to a string (true is displayed as 'true' false as 'false')
*
* - If an integer / float is returned, we simply return it
*
* @param mixed $value The initial returned value, before post process. If set to null, success response is returned.
* @param bool|string $apiModule The API module that was called
* @param bool|string $apiMethod The API method that was called
* @return mixed Usually a string, but can still be a PHP data structure if the format requested is 'original'
*/
public function getResponse($value = null, $apiModule = false, $apiMethod = false)
{
$this->apiModule = $apiModule;
$this->apiMethod = $apiMethod;
$this->sendHeaderIfEnabled();
// when null or void is returned from the api call, we handle it as a successful operation
if (!isset($value)) {
if (ob_get_contents()) {
return null;
}
return $this->apiRenderer->renderSuccess('ok');
}
// If the returned value is an object DataTable we
// apply the set of generic filters if asked in the URL
// and we render the DataTable according to the format specified in the URL
if ($value instanceof DataTableInterface) {
return $this->handleDataTable($value);
}
// Case an array is returned from the API call, we convert it to the requested format
// - if calling from inside the application (format = original)
// => the data stays unchanged (ie. a standard php array or whatever data structure)
// - if any other format is requested, we have to convert this data structure (which we assume
// to be an array) to a DataTable in order to apply the requested DataTable_Renderer (for example XML)
if (is_array($value)) {
return $this->handleArray($value);
}
if (is_object($value)) {
return $this->apiRenderer->renderObject($value);
}
if (is_resource($value)) {
return $this->apiRenderer->renderResource($value);
}
return $this->apiRenderer->renderScalar($value);
}
/**
* Returns an error $message in the requested $format
*
* @param Exception|\Throwable $e
* @throws Exception
* @return string
*/
public function getResponseException($e)
{
$e = $this->decorateExceptionWithDebugTrace($e);
$message = $this->formatExceptionMessage($e);
if ($this->sendHeader
&& $e instanceof HttpCodeException
&& $e->getCode() > 0
) {
http_response_code($e->getCode());
}
$this->sendHeaderIfEnabled();
return $this->apiRenderer->renderException($message, $e);
}
/**
* @param Exception|\Throwable $e
* @return Exception
*/
private function decorateExceptionWithDebugTrace($e)
{
// If we are in tests, show full backtrace
if (defined('PIWIK_PATH_TEST_TO_ROOT')) {
if ($this->shouldPrintBacktrace) {
$message = $e->getMessage() . " in \n " . $e->getFile() . ":" . $e->getLine() . " \n " . $e->getTraceAsString();
} else {
$message = $e->getMessage() . "\n \n --> To temporarily debug this error further, set const PIWIK_PRINT_ERROR_BACKTRACE=true; in index.php";
}
return new Exception($message);
}
return $e;
}
/**
* @param Exception|\Throwable $exception
* @return string
*/
private function formatExceptionMessage($exception)
{
$message = ExceptionToTextProcessor::getMessageAndWholeBacktrace($exception, $this->shouldPrintBacktrace);
if ($exception instanceof \Piwik\Exception\Exception && $exception->isHtmlMessage() && Request::isRootRequestApiRequest()) {
$message = strip_tags(str_replace('<br />', PHP_EOL, $message));
}
return Renderer::formatValueXml($message);
}
private function handleDataTable(DataTableInterface $datatable)
{
if ($this->postProcessDataTable) {
$postProcessor = new DataTablePostProcessor($this->apiModule, $this->apiMethod, $this->request);
$datatable = $postProcessor->process($datatable);
}
return $this->apiRenderer->renderDataTable($datatable);
}
private function handleArray($array)
{
$firstArray = null;
$firstKey = null;
if (!empty($array)) {
$firstArray = reset($array);
$firstKey = key($array);
}
$isAssoc = !empty($firstArray) && is_numeric($firstKey) && is_array($firstArray) && count(array_filter(array_keys($firstArray), 'is_string'));
if (is_numeric($firstKey)) {
$columns = Common::getRequestVar('filter_column', false, 'array', $this->request);
$pattern = Common::getRequestVar('filter_pattern', '', 'string', $this->request);
if ($columns != array(false) && $pattern !== '') {
$pattern = new Pattern(new DataTable(), $columns, $pattern);
$array = $pattern->filterArray($array);
}
$limit = Common::getRequestVar('filter_limit', -1, 'integer', $this->request);
$offset = Common::getRequestVar('filter_offset', '0', 'integer', $this->request);
if ($limit >= 0 || $offset > 0) {
if ($limit < 0) {
$limit = null; // make sure to return all results from offset
}
$array = array_slice($array, $offset, $limit, $preserveKeys = false);
}
}
if ($isAssoc) {
$hideColumns = Common::getRequestVar('hideColumns', '', 'string', $this->request);
$showColumns = Common::getRequestVar('showColumns', '', 'string', $this->request);
if ($hideColumns !== '' || $showColumns !== '') {
$columnDelete = new ColumnDelete(new DataTable(), $hideColumns, $showColumns);
$array = $columnDelete->filter($array);
}
}
return $this->apiRenderer->renderArray($array);
}
private function sendHeaderIfEnabled()
{
if ($this->sendHeader) {
$this->apiRenderer->sendHeader();
}
}
}

768
matomo/core/Access.php Normal file
View file

@ -0,0 +1,768 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
use Exception;
use Piwik\Access\CapabilitiesProvider;
use Piwik\API\Request;
use Piwik\Access\RolesProvider;
use Piwik\Container\StaticContainer;
use Piwik\Plugins\SitesManager\API as SitesManagerApi;
use Piwik\Session\SessionAuth;
/**
* Singleton that manages user access to Piwik resources.
*
* To check whether a user has access to a resource, use one of the {@link Piwik Piwik::checkUser...}
* methods.
*
* In Piwik there are four different access levels:
*
* - **no access**: Users with this access level cannot view the resource.
* - **view access**: Users with this access level can view the resource, but cannot modify it.
* - **admin access**: Users with this access level can view and modify the resource.
* - **Super User access**: Only the Super User has this access level. It means the user can do
* whatever they want.
*
* Super user access is required to set some configuration options.
* All other options are specific to the user or to a website.
*
* Access is granted per website. Uses with access for a website can view all
* data associated with that website.
*
*/
class Access
{
/**
* Array of idsites available to the current user, indexed by permission level
* @see getSitesIdWith*()
*
* @var array
*/
protected $idsitesByAccess = null;
/**
* Login of the current user
*
* @var string
*/
protected $login = null;
/**
* token_auth of the current user
*
* @var string
*/
protected $token_auth = null;
/**
* Defines if the current user is the Super User
* @see hasSuperUserAccess()
*
* @var bool
*/
protected $hasSuperUserAccess = false;
/**
* Authentification object (see Auth)
*
* @var Auth
*/
private $auth = null;
/**
* Gets the singleton instance. Creates it if necessary.
*
* @return self
*/
public static function getInstance()
{
return StaticContainer::get('Piwik\Access');
}
/**
* @var CapabilitiesProvider
*/
protected $capabilityProvider;
/**
* @var RolesProvider
*/
private $roleProvider;
/**
* Constructor
*/
public function __construct(RolesProvider $roleProvider = null, CapabilitiesProvider $capabilityProvider = null)
{
if (!isset($roleProvider)) {
$roleProvider = StaticContainer::get('Piwik\Access\RolesProvider');
}
if (!isset($capabilityProvider)) {
$capabilityProvider = StaticContainer::get('Piwik\Access\CapabilitiesProvider');
}
$this->roleProvider = $roleProvider;
$this->capabilityProvider = $capabilityProvider;
$this->resetSites();
}
private function resetSites()
{
$this->idsitesByAccess = array(
'view' => array(),
'write' => array(),
'admin' => array(),
'superuser' => array(),
);
}
/**
* Loads the access levels for the current user.
*
* Calls the authentication method to try to log the user in the system.
* If the user credentials are not correct we don't load anything.
* If the login/password is correct the user is either the SuperUser or a normal user.
* We load the access levels for this user for all the websites.
*
* @param null|Auth $auth Auth adapter
* @return bool true on success, false if reloading access failed (when auth object wasn't specified and user is not enforced to be Super User)
*/
public function reloadAccess(Auth $auth = null)
{
$this->resetSites();
if (isset($auth)) {
$this->auth = $auth;
}
if ($this->hasSuperUserAccess()) {
$this->makeSureLoginNameIsSet();
return true;
}
$this->token_auth = null;
$this->login = null;
// if the Auth wasn't set, we may be in the special case of setSuperUser(), otherwise we fail TODO: docs + review
if (!isset($this->auth)) {
return false;
}
$result = null;
$forceApiSessionPost = Common::getRequestVar('force_api_session', 0, 'int', $_POST);
$forceApiSessionGet = Common::getRequestVar('force_api_session', 0, 'int', $_GET);
$isApiRequest = Piwik::getModule() === 'API' && (Piwik::getAction() === 'index' || !Piwik::getAction());
$apiMethod = Request::getMethodIfApiRequest(null);
$isGetApiRequest = !empty($apiMethod) && 1 === substr_count($apiMethod, '.') && strpos($apiMethod, '.get') > 0;
if (($forceApiSessionPost && $isApiRequest) || ($forceApiSessionGet && $isApiRequest && $isGetApiRequest)) {
$request = ($forceApiSessionGet && $isApiRequest && $isGetApiRequest) ? $_GET : $_POST;
$tokenAuth = Common::getRequestVar('token_auth', '', 'string', $request);
Session::start();
$auth = StaticContainer::get(SessionAuth::class);
$auth->setTokenAuth($tokenAuth);
$result = $auth->authenticate();
// Note: We do not post a failed login event at this point on purpose
// If using the SessionAuth doesn't work, the FrontController will try to reload the Auth using
// the token_auth only. If that works everything is "fine" and the `force_api_session` parameter was
// unneeded. If that fails as well it will trigger the failed login event
// See FrontController::init() or Request::reloadAuthUsingTokenAuth()
Session::close();
// if not successful, we will fallback to regular auth
}
// access = array ( idsite => accessIdSite, idsite2 => accessIdSite2)
if (!$result || !$result->wasAuthenticationSuccessful()) {
$result = $this->auth->authenticate();
}
if (!$result->wasAuthenticationSuccessful()) {
return false;
}
$this->login = $result->getIdentity();
$this->token_auth = $result->getTokenAuth();
// case the superUser is logged in
if ($result->hasSuperUserAccess()) {
$this->setSuperUserAccess(true);
}
return true;
}
public function getRawSitesWithSomeViewAccess($login)
{
$sql = self::getSqlAccessSite("access, t2.idsite");
return Db::fetchAll($sql, $login);
}
/**
* Returns the SQL query joining sites and access table for a given login
*
* @param string $select Columns or expression to SELECT FROM table, eg. "MIN(ts_created)"
* @return string SQL query
*/
public static function getSqlAccessSite($select)
{
$access = Common::prefixTable('access');
$siteTable = Common::prefixTable('site');
return "SELECT " . $select . " FROM " . $access . " as t1
JOIN " . $siteTable . " as t2 USING (idsite) WHERE login = ?";
}
/**
* Make sure a login name is set
*
* @return true
*/
protected function makeSureLoginNameIsSet()
{
if (empty($this->login)) {
// flag to force non empty login so Super User is not mistaken for anonymous
$this->login = 'super user was set';
}
}
protected function loadSitesIfNeeded()
{
if ($this->hasSuperUserAccess) {
if (empty($this->idsitesByAccess['superuser'])) {
try {
$api = SitesManagerApi::getInstance();
$allSitesId = $api->getAllSitesId();
} catch (\Exception $e) {
$allSitesId = array();
}
$this->idsitesByAccess['superuser'] = $allSitesId;
}
} elseif (isset($this->login)) {
if (empty($this->idsitesByAccess['view'])
&& empty($this->idsitesByAccess['write'])
&& empty($this->idsitesByAccess['admin'])
) {
// we join with site in case there are rows in access for an idsite that doesn't exist anymore
// (backward compatibility ; before we deleted the site without deleting rows in _access table)
$accessRaw = $this->getRawSitesWithSomeViewAccess($this->login);
foreach ($accessRaw as $access) {
$accessType = $access['access'];
$this->idsitesByAccess[$accessType][] = $access['idsite'];
if ($this->roleProvider->isValidRole($accessType)) {
foreach ($this->capabilityProvider->getAllCapabilities() as $capability) {
if ($capability->hasRoleCapability($accessType)) {
// we automatically add this capability
if (!isset($this->idsitesByAccess[$capability->getId()])) {
$this->idsitesByAccess[$capability->getId()] = array();
}
$this->idsitesByAccess[$capability->getId()][] = $access['idsite'];
}
}
}
}
/**
* Triggered after the initial access levels and permissions for the current user are loaded. Use this
* event to modify the current user's permissions (for example, making sure every user has view access
* to a specific site).
*
* **Example**
*
* function (&$idsitesByAccess, $login) {
* if ($login == 'somespecialuser') {
* return;
* }
*
* $idsitesByAccess['view'][] = $mySpecialIdSite;
* }
*
* @param array[] &$idsitesByAccess The current user's access levels for individual sites. Maps role and
* capability IDs to list of site IDs, eg:
*
* ```
* [
* 'view' => [1, 2, 3],
* 'write' => [4, 5],
* 'admin' => [],
* ]
* ```
* @param string $login The current user's login.
*/
Piwik::postEvent('Access.modifyUserAccess', [&$this->idsitesByAccess, $this->login]);
}
}
}
/**
* We bypass the normal auth method and give the current user Super User rights.
* This should be very carefully used.
*
* @param bool $bool
*/
public function setSuperUserAccess($bool = true)
{
$this->hasSuperUserAccess = (bool) $bool;
if ($bool) {
$this->makeSureLoginNameIsSet();
} else {
$this->resetSites();
}
}
/**
* Returns true if the current user is logged in as the Super User
*
* @return bool
*/
public function hasSuperUserAccess()
{
return $this->hasSuperUserAccess;
}
/**
* Returns the current user login
*
* @return string|null
*/
public function getLogin()
{
return $this->login;
}
/**
* Returns the token_auth used to authenticate this user in the API
*
* @return string|null
*/
public function getTokenAuth()
{
return $this->token_auth;
}
/**
* Returns an array of ID sites for which the user has at least a VIEW access.
* Which means VIEW OR WRITE or ADMIN or SUPERUSER.
*
* @return array Example if the user is ADMIN for 4
* and has VIEW access for 1 and 7, it returns array(1, 4, 7);
*/
public function getSitesIdWithAtLeastViewAccess()
{
$this->loadSitesIfNeeded();
return array_unique(array_merge(
$this->idsitesByAccess['view'],
$this->idsitesByAccess['write'],
$this->idsitesByAccess['admin'],
$this->idsitesByAccess['superuser'])
);
}
/**
* Returns an array of ID sites for which the user has at least a WRITE access.
* Which means WRITE or ADMIN or SUPERUSER.
*
* @return array Example if the user is WRITE for 4 and 8
* and has VIEW access for 1 and 7, it returns array(4, 8);
*/
public function getSitesIdWithAtLeastWriteAccess()
{
$this->loadSitesIfNeeded();
return array_unique(array_merge(
$this->idsitesByAccess['write'],
$this->idsitesByAccess['admin'],
$this->idsitesByAccess['superuser'])
);
}
/**
* Returns an array of ID sites for which the user has an ADMIN access.
*
* @return array Example if the user is ADMIN for 4 and 8
* and has VIEW access for 1 and 7, it returns array(4, 8);
*/
public function getSitesIdWithAdminAccess()
{
$this->loadSitesIfNeeded();
return array_unique(array_merge(
$this->idsitesByAccess['admin'],
$this->idsitesByAccess['superuser'])
);
}
/**
* Returns an array of ID sites for which the user has a VIEW access only.
*
* @return array Example if the user is ADMIN for 4
* and has VIEW access for 1 and 7, it returns array(1, 7);
* @see getSitesIdWithAtLeastViewAccess()
*/
public function getSitesIdWithViewAccess()
{
$this->loadSitesIfNeeded();
return $this->idsitesByAccess['view'];
}
/**
* Returns an array of ID sites for which the user has a WRITE access only.
*
* @return array Example if the user is ADMIN for 4
* and has WRITE access for 1 and 7, it returns array(1, 7);
* @see getSitesIdWithAtLeastWriteAccess()
*/
public function getSitesIdWithWriteAccess()
{
$this->loadSitesIfNeeded();
return $this->idsitesByAccess['write'];
}
/**
* Throws an exception if the user is not the SuperUser
*
* @throws \Piwik\NoAccessException
*/
public function checkUserHasSuperUserAccess()
{
if (!$this->hasSuperUserAccess()) {
$this->throwNoAccessException(Piwik::translate('General_ExceptionPrivilege', array("'superuser'")));
}
}
/**
* Returns `true` if the current user has admin access to at least one site.
*
* @return bool
*/
public function isUserHasSomeWriteAccess()
{
if ($this->hasSuperUserAccess()) {
return true;
}
$idSitesAccessible = $this->getSitesIdWithAtLeastWriteAccess();
return count($idSitesAccessible) > 0;
}
/**
* Returns `true` if the current user has admin access to at least one site.
*
* @return bool
*/
public function isUserHasSomeAdminAccess()
{
if ($this->hasSuperUserAccess()) {
return true;
}
$idSitesAccessible = $this->getSitesIdWithAdminAccess();
return count($idSitesAccessible) > 0;
}
/**
* If the user doesn't have an WRITE access for at least one website, throws an exception
*
* @throws \Piwik\NoAccessException
*/
public function checkUserHasSomeWriteAccess()
{
if (!$this->isUserHasSomeWriteAccess()) {
$this->throwNoAccessException(Piwik::translate('General_ExceptionPrivilegeAtLeastOneWebsite', array('write')));
}
}
/**
* If the user doesn't have an ADMIN access for at least one website, throws an exception
*
* @throws \Piwik\NoAccessException
*/
public function checkUserHasSomeAdminAccess()
{
if (!$this->isUserHasSomeAdminAccess()) {
$this->throwNoAccessException(Piwik::translate('General_ExceptionPrivilegeAtLeastOneWebsite', array('admin')));
}
}
/**
* If the user doesn't have any view permission, throw exception
*
* @throws \Piwik\NoAccessException
*/
public function checkUserHasSomeViewAccess()
{
if ($this->hasSuperUserAccess()) {
return;
}
$idSitesAccessible = $this->getSitesIdWithAtLeastViewAccess();
if (count($idSitesAccessible) == 0) {
$this->throwNoAccessException(Piwik::translate('General_ExceptionPrivilegeAtLeastOneWebsite', array('view')));
}
}
/**
* This method checks that the user has ADMIN access for the given list of websites.
* If the user doesn't have ADMIN access for at least one website of the list, we throw an exception.
*
* @param int|array $idSites List of ID sites to check
* @throws \Piwik\NoAccessException If for any of the websites the user doesn't have an ADMIN access
*/
public function checkUserHasAdminAccess($idSites)
{
if ($this->hasSuperUserAccess()) {
return;
}
$idSites = $this->getIdSites($idSites);
$idSitesAccessible = $this->getSitesIdWithAdminAccess();
foreach ($idSites as $idsite) {
if (!in_array($idsite, $idSitesAccessible)) {
$this->throwNoAccessException(Piwik::translate('General_ExceptionPrivilegeAccessWebsite', array("'admin'", $idsite)));
}
}
}
/**
* This method checks that the user has VIEW or ADMIN access for the given list of websites.
* If the user doesn't have VIEW or ADMIN access for at least one website of the list, we throw an exception.
*
* @param int|array|string $idSites List of ID sites to check (integer, array of integers, string comma separated list of integers)
* @throws \Piwik\NoAccessException If for any of the websites the user doesn't have an VIEW or ADMIN access
*/
public function checkUserHasViewAccess($idSites)
{
if ($this->hasSuperUserAccess()) {
return;
}
$idSites = $this->getIdSites($idSites);
$idSitesAccessible = $this->getSitesIdWithAtLeastViewAccess();
foreach ($idSites as $idsite) {
if (!in_array($idsite, $idSitesAccessible)) {
$this->throwNoAccessException(Piwik::translate('General_ExceptionPrivilegeAccessWebsite', array("'view'", $idsite)));
}
}
}
/**
* This method checks that the user has VIEW or ADMIN access for the given list of websites.
* If the user doesn't have VIEW or ADMIN access for at least one website of the list, we throw an exception.
*
* @param int|array|string $idSites List of ID sites to check (integer, array of integers, string comma separated list of integers)
* @throws \Piwik\NoAccessException If for any of the websites the user doesn't have an VIEW or ADMIN access
*/
public function checkUserHasWriteAccess($idSites)
{
if ($this->hasSuperUserAccess()) {
return;
}
$idSites = $this->getIdSites($idSites);
$idSitesAccessible = $this->getSitesIdWithAtLeastWriteAccess();
foreach ($idSites as $idsite) {
if (!in_array($idsite, $idSitesAccessible)) {
$this->throwNoAccessException(Piwik::translate('General_ExceptionPrivilegeAccessWebsite', array("'write'", $idsite)));
}
}
}
public function checkUserIsNotAnonymous()
{
if ($this->hasSuperUserAccess()) {
return;
}
if (Piwik::isUserIsAnonymous()) {
$this->throwNoAccessException(Piwik::translate('General_YouMustBeLoggedIn'));
}
}
private function getSitesIdWithCapability($capability)
{
if (!empty($this->idsitesByAccess[$capability])) {
return $this->idsitesByAccess[$capability];
}
return array();
}
public function checkUserHasCapability($idSites, $capability)
{
if ($this->hasSuperUserAccess()) {
return;
}
$idSites = $this->getIdSites($idSites);
$idSitesAccessible = $this->getSitesIdWithCapability($capability);
foreach ($idSites as $idsite) {
if (!in_array($idsite, $idSitesAccessible)) {
$this->throwNoAccessException(Piwik::translate('General_ExceptionCapabilityAccessWebsite', array("'" . $capability ."'", $idsite)));
}
}
// a capability applies only when the user also has at least view access
$this->checkUserHasViewAccess($idSites);
}
/**
* @param int|array|string $idSites
* @return array
* @throws \Piwik\NoAccessException
*/
protected function getIdSites($idSites)
{
if ($idSites === 'all') {
$idSites = $this->getSitesIdWithAtLeastViewAccess();
}
$idSites = Site::getIdSitesFromIdSitesString($idSites);
if (empty($idSites)) {
$this->throwNoAccessException("The parameter 'idSite=' is missing from the request.");
}
return $idSites;
}
/**
* Executes a callback with superuser privileges, making sure those privileges are rescinded
* before this method exits. Privileges will be rescinded even if an exception is thrown.
*
* @param callback $function The callback to execute. Should accept no arguments.
* @return mixed The result of `$function`.
* @throws Exception rethrows any exceptions thrown by `$function`.
* @api
*/
public static function doAsSuperUser($function)
{
$isSuperUser = self::getInstance()->hasSuperUserAccess();
if ($isSuperUser) {
return $function();
}
$access = self::getInstance();
$login = $access->getLogin();
$shouldResetLogin = empty($login); // make sure to reset login if a login was set by "makeSureLoginNameIsSet()"
$access->setSuperUserAccess(true);
try {
$result = $function();
} catch (\Throwable $ex) {
$access->setSuperUserAccess($isSuperUser);
if ($shouldResetLogin) {
$access->login = null;
}
throw $ex;
}
if ($shouldResetLogin) {
$access->login = null;
}
$access->setSuperUserAccess($isSuperUser);
return $result;
}
/**
* Returns the level of access the current user has to the given site.
*
* @param int $idSite The site to check.
* @return string The access level, eg, 'view', 'admin', 'noaccess'.
*/
public function getRoleForSite($idSite)
{
if ($this->hasSuperUserAccess
|| in_array($idSite, $this->getSitesIdWithAdminAccess())
) {
return 'admin';
}
if (in_array($idSite, $this->getSitesIdWithWriteAccess())) {
return 'write';
}
if (in_array($idSite, $this->getSitesIdWithViewAccess())) {
return 'view';
}
return 'noaccess';
}
/**
* Returns the capabilities the current user has for a given site.
*
* @param int $idSite The site to check.
* @return string[] The capabilities the user has.
*/
public function getCapabilitiesForSite($idSite)
{
$result = [];
foreach ($this->capabilityProvider->getAllCapabilityIds() as $capabilityId) {
if (empty($this->idsitesByAccess[$capabilityId])) {
continue;
}
if (in_array($idSite, $this->idsitesByAccess[$capabilityId])) {
$result[] = $capabilityId;
}
}
return $result;
}
/**
* Throw a NoAccessException with the given message, or a more generic 'You need to log in' message if the
* user is not currently logged in (e.g. if session has expired).
*
* @param $message
* @throws NoAccessException
*/
private function throwNoAccessException($message)
{
if (Piwik::isUserIsAnonymous() && !Request::isRootRequestApiRequest()) {
$message = Piwik::translate('General_YouMustBeLoggedIn');
// Try to detect whether user was previously logged in so that we can display a different message
$referrer = Url::getReferrer();
$matomoUrl = SettingsPiwik::getPiwikUrl();
if ($referrer && $matomoUrl && Url::isValidHost(Url::getHostFromUrl($referrer)) &&
strpos($referrer, $matomoUrl) === 0
) {
$message = Piwik::translate('General_YourSessionHasExpired');
}
}
throw new NoAccessException($message);
}
/**
* Returns true if the current user is logged in or not.
*
* @return bool
*/
public function isUserLoggedIn()
{
return !empty($this->login);
}
}

141
matomo/core/Access/CapabilitiesProvider.php Normal file
View file

@ -0,0 +1,141 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Access;
use Exception;
use Piwik\CacheId;
use Piwik\Piwik;
use Piwik\Cache as PiwikCache;
class CapabilitiesProvider
{
/**
* @return Capability[]
* @throws Exception
*/
public function getAllCapabilities(): array
{
$cacheId = CacheId::siteAware(CacheId::languageAware('Capabilities'));
$cache = PiwikCache::getTransientCache();
if (!$cache->contains($cacheId)) {
$capabilities = [];
/**
* Triggered to add new capabilities.
*
* **Example**
*
* public function addCapabilities(&$capabilities)
* {
* $capabilities[] = new MyNewCapability();
* }
*
* @param Capability[] $reports An array of reports
* @internal
*/
Piwik::postEvent('Access.Capability.addCapabilities', array(&$capabilities));
/**
* Triggered to filter / restrict capabilities.
*
* **Example**
*
* public function filterCapabilities(&$capabilities)
* {
* foreach ($capabilities as $index => $capability) {
* if ($capability->getId() === 'tagmanager_write') {}
* unset($capabilities[$index]); // remove the given capability
* }
* }
* }
*
* @param Capability[] $reports An array of reports
* @internal
*/
Piwik::postEvent('Access.Capability.filterCapabilities', array(&$capabilities));
$capabilities = \array_values($capabilities);
$this->checkCapabilityIds($capabilities);
$cache->save($cacheId, $capabilities);
return $capabilities;
}
return $cache->fetch($cacheId);
}
/**
* @param $capabilityId
* @return Capability|null
* @throws Exception
*/
public function getCapability(string $capabilityId): ?Capability
{
foreach ($this->getAllCapabilities() as $capability) {
if ($capabilityId === $capability->getId()) {
return $capability;
}
}
return null;
}
/**
* @return string[]
* @throws Exception
*/
public function getAllCapabilityIds(): array
{
$ids = array();
foreach ($this->getAllCapabilities() as $capability) {
$ids[] = $capability->getId();
}
return $ids;
}
/**
* @param $capabilityId
* @return bool
* @throws Exception
*/
public function isValidCapability($capabilityId): bool
{
$capabilities = $this->getAllCapabilityIds();
return \in_array($capabilityId, $capabilities, true);
}
/**
* @param $capabilityId
* @throws Exception
*/
public function checkValidCapability($capabilityId): void
{
if (!$this->isValidCapability($capabilityId)) {
$capabilities = $this->getAllCapabilityIds();
throw new \Exception(Piwik::translate("UsersManager_ExceptionAccessValues", implode(", ", $capabilities)));
}
}
/**
* @param Capability[] $capabilities
* @throws Exception
*/
private function checkCapabilityIds(array $capabilities): void
{
foreach ($capabilities as $capability) {
$id = $capability->getId();
if (preg_match('/[^a-zA-Z0-9_-]/', $id)) {
throw new \Exception("Capability with invalid ID found: '$id'. Valid characters are 'a-zA-Z0-9_-'.");
}
}
}
}

29
matomo/core/Access/Capability.php Normal file
View file

@ -0,0 +1,29 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Access;
abstract class Capability
{
abstract public function getId(): string;
abstract public function getName(): string;
abstract public function getCategory(): string;
abstract public function getDescription(): string;
abstract public function getIncludedInRoles(): array;
public function getHelpUrl(): string
{
return '';
}
public function hasRoleCapability(string $idRole): bool
{
return \in_array($idRole, $this->getIncludedInRoles(), true);
}
}

22
matomo/core/Access/Role.php Normal file
View file

@ -0,0 +1,22 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Access;
abstract class Role
{
abstract public function getName(): string;
abstract public function getId(): string;
abstract public function getDescription(): string;
public function getHelpUrl(): string
{
return '';
}
}

40
matomo/core/Access/Role/Admin.php Normal file
View file

@ -0,0 +1,40 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Access\Role;
use Piwik\Access\Role;
use Piwik\Piwik;
class Admin extends Role
{
public const ID = 'admin';
public function getName(): string
{
return Piwik::translate('UsersManager_PrivAdmin');
}
public function getId(): string
{
return self::ID;
}
public function getDescription(): string
{
return Piwik::translate('UsersManager_PrivAdminDescription', array(
Piwik::translate('UsersManager_PrivWrite')
));
}
public function getHelpUrl(): string
{
return 'https://matomo.org/faq/general/faq_69/';
}
}

39
matomo/core/Access/Role/View.php Normal file
View file

@ -0,0 +1,39 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Access\Role;
use Piwik\Access\Role;
use Piwik\Piwik;
class View extends Role
{
public const ID = 'view';
public function getName(): string
{
return Piwik::translate('UsersManager_PrivView');
}
public function getId(): string
{
return self::ID;
}
public function getDescription(): string
{
return Piwik::translate('UsersManager_PrivViewDescription');
}
public function getHelpUrl(): string
{
return 'https://matomo.org/faq/general/faq_70/';
}
}

38
matomo/core/Access/Role/Write.php Normal file
View file

@ -0,0 +1,38 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Access\Role;
use Piwik\Access\Role;
use Piwik\Piwik;
class Write extends Role
{
public const ID = 'write';
public function getName(): string
{
return Piwik::translate('UsersManager_PrivWrite');
}
public function getId(): string
{
return self::ID;
}
public function getDescription(): string
{
return Piwik::translate('UsersManager_PrivWriteDescription');
}
public function getHelpUrl(): string
{
return 'https://matomo.org/faq/general/faq_26910';
}
}

66
matomo/core/Access/RolesProvider.php Normal file
View file

@ -0,0 +1,66 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Access;
use Piwik\Access\Role\Admin;
use Piwik\Access\Role\View;
use Piwik\Access\Role\Write;
use Piwik\Piwik;
use Exception;
class RolesProvider
{
/**
* @return Role[]
*/
public function getAllRoles(): array
{
return array(
new View(),
new Write(),
new Admin()
);
}
/**
* Returns the list of the existing Access level.
* Useful when a given API method requests a given access Level.
* We first check that the required access level exists.
*
* @return string[]
*/
public function getAllRoleIds(): array
{
$ids = array();
foreach ($this->getAllRoles() as $role) {
$ids[] = $role->getId();
}
return $ids;
}
public function isValidRole(string $roleId): bool
{
$roles = $this->getAllRoleIds();
return \in_array($roleId, $roles, true);
}
/**
* @param $roleId
* @throws Exception
*/
public function checkValidRole(string $roleId): void
{
if (!$this->isValidRole($roleId)) {
$roles = $this->getAllRoleIds();
throw new Exception(Piwik::translate("UsersManager_ExceptionAccessValues", [implode(", ", $roles), $roleId]));
}
}
}

252
matomo/core/Application/Environment.php Normal file
View file

@ -0,0 +1,252 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application;
use DI\Container;
use Piwik\Application\Kernel\EnvironmentValidator;
use Piwik\Application\Kernel\GlobalSettingsProvider;
use Piwik\Application\Kernel\PluginList;
use Piwik\Container\ContainerFactory;
use Piwik\Container\StaticContainer;
use Piwik\Piwik;
/**
* Encapsulates Piwik environment setup and access.
*
* The Piwik environment consists of two main parts: the kernel and the DI container.
*
* The 'kernel' is the core part of Piwik that cannot be modified / extended through the DI container.
* It includes components that are required to create the DI container.
*
* Currently the only objects in the 'kernel' are a GlobalSettingsProvider object and a
* PluginList object. The GlobalSettingsProvider object is required for the current PluginList
* implementation and for checking whether Development mode is enabled. The PluginList is
* needed in order to determine what plugins are activated, since plugins can provide their
* own DI configuration.
*
* The DI container contains every other Piwik object, including the Plugin\Manager,
* plugin API instances, dependent services, etc. Plugins and users can override/extend
* the objects in this container.
*
* NOTE: DI support in Piwik is currently a work in process; not everything is currently
* stored in the DI container, but we are working towards this.
*/
class Environment
{
/**
* @internal
* @var EnvironmentManipulator
*/
private static $globalEnvironmentManipulator = null;
/**
* @var string
*/
private $environment;
/**
* @var array
*/
private $definitions;
/**
* @var Container
*/
private $container;
/**
* @var GlobalSettingsProvider
*/
private $globalSettingsProvider;
/**
* @var PluginList
*/
private $pluginList;
/**
* @param string $environment
* @param array $definitions
*/
public function __construct($environment, array $definitions = array())
{
$this->environment = $environment;
$this->definitions = $definitions;
}
public function getEnvironmentName()
{
return $this->environment;
}
/**
* Initializes the kernel globals and DI container.
*/
public function init()
{
$this->invokeBeforeContainerCreatedHook();
$this->container = $this->createContainer();
$this->container->set(self::class, $this);
StaticContainer::push($this->container);
$this->validateEnvironment();
$this->invokeEnvironmentBootstrappedHook();
Piwik::postEvent('Environment.bootstrapped'); // this event should be removed eventually
}
/**
* Destroys an environment. MUST be called when embedding environments.
*/
public function destroy()
{
StaticContainer::pop();
}
/**
* Returns the DI container. All Piwik objects for a specific Piwik instance should be stored
* in this container.
*
* @return Container
*/
public function getContainer()
{
return $this->container;
}
/**
* @link http://php-di.org/doc/container-configuration.html
*/
private function createContainer()
{
$pluginList = $this->getPluginListCached();
$settings = $this->getGlobalSettingsCached();
$extraDefinitions = $this->getExtraDefinitionsFromManipulators();
$definitions = array_merge(StaticContainer::getDefinitions(), $extraDefinitions, array($this->definitions));
$environments = array($this->environment);
$environments = array_merge($environments, $this->getExtraEnvironmentsFromManipulators());
$containerFactory = new ContainerFactory($pluginList, $settings, $environments, $definitions);
return $containerFactory->create();
}
protected function getGlobalSettingsCached()
{
if ($this->globalSettingsProvider === null) {
$original = $this->getGlobalSettings();
$globalSettingsProvider = $this->getGlobalSettingsProviderOverride($original);
$this->globalSettingsProvider = $globalSettingsProvider ?: $original;
}
return $this->globalSettingsProvider;
}
protected function getPluginListCached()
{
if ($this->pluginList === null) {
$pluginList = $this->getPluginListOverride();
$this->pluginList = $pluginList ?: $this->getPluginList();
}
return $this->pluginList;
}
/**
* Returns the kernel global GlobalSettingsProvider object. Derived classes can override this method
* to provide a different implementation.
*
* @return null|GlobalSettingsProvider
*/
protected function getGlobalSettings()
{
return new GlobalSettingsProvider();
}
/**
* Returns the kernel global PluginList object. Derived classes can override this method to
* provide a different implementation.
*
* @return PluginList
*/
protected function getPluginList()
{
// TODO: in tracker should only load tracker plugins. can't do properly until tracker entrypoint is encapsulated.
return new PluginList($this->getGlobalSettingsCached());
}
private function validateEnvironment()
{
/** @var EnvironmentValidator $validator */
$validator = $this->container->get('Piwik\Application\Kernel\EnvironmentValidator');
$validator->validate();
}
/**
* @param EnvironmentManipulator $manipulator
* @internal
*/
public static function setGlobalEnvironmentManipulator(EnvironmentManipulator $manipulator)
{
self::$globalEnvironmentManipulator = $manipulator;
}
private function getGlobalSettingsProviderOverride(GlobalSettingsProvider $original)
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->makeGlobalSettingsProvider($original);
} else {
return null;
}
}
private function invokeBeforeContainerCreatedHook()
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->beforeContainerCreated();
}
}
private function getExtraDefinitionsFromManipulators()
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->getExtraDefinitions();
} else {
return array();
}
}
private function invokeEnvironmentBootstrappedHook()
{
if (self::$globalEnvironmentManipulator) {
self::$globalEnvironmentManipulator->onEnvironmentBootstrapped();
}
}
private function getExtraEnvironmentsFromManipulators()
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->getExtraEnvironments();
} else {
return array();
}
}
private function getPluginListOverride()
{
if (self::$globalEnvironmentManipulator) {
return self::$globalEnvironmentManipulator->makePluginList($this->getGlobalSettingsCached());
} else {
return null;
}
}
}

59
matomo/core/Application/EnvironmentManipulator.php Normal file
View file

@ -0,0 +1,59 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application;
use Piwik\Application\Kernel\GlobalSettingsProvider;
use Piwik\Application\Kernel\PluginList;
/**
* Used to manipulate Environment instances before the container is created.
* Only used by the testing environment setup code, shouldn't be used anywhere
* else.
*/
interface EnvironmentManipulator
{
/**
* Create a custom GlobalSettingsProvider kernel object, overriding the default behavior.
*
* @return GlobalSettingsProvider
*/
public function makeGlobalSettingsProvider(GlobalSettingsProvider $original);
/**
* Create a custom PluginList kernel object, overriding the default behavior.
*
* @param GlobalSettingsProvider $globalSettingsProvider
* @return PluginList
*/
public function makePluginList(GlobalSettingsProvider $globalSettingsProvider);
/**
* Invoked before the container is created.
*/
public function beforeContainerCreated();
/**
* Return an array of definition arrays that override DI config specified in PHP config files.
*
* @return array[]
*/
public function getExtraDefinitions();
/**
* Invoked after the container is created and the environment is considered bootstrapped.
*/
public function onEnvironmentBootstrapped();
/**
* Return an array of environment names to apply after the normal environment.
*
* @return string[]
*/
public function getExtraEnvironments();
}

151
matomo/core/Application/Kernel/EnvironmentValidator.php Normal file
View file

@ -0,0 +1,151 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application\Kernel;
use Piwik\Common;
use Piwik\Exception\NotYetInstalledException;
use Piwik\Filechecks;
use Piwik\Piwik;
use Piwik\SettingsPiwik;
use Piwik\SettingsServer;
use Piwik\Translation\Translator;
/**
* Validates the Piwik environment. This includes making sure the required config files
* are present, and triggering the correct behaviour if otherwise.
*/
class EnvironmentValidator
{
/**
* @var GlobalSettingsProvider
*/
protected $settingsProvider;
/**
* @var Translator
*/
protected $translator;
public function __construct(GlobalSettingsProvider $settingsProvider, Translator $translator)
{
$this->settingsProvider = $settingsProvider;
$this->translator = $translator;
}
public function validate()
{
$this->checkConfigFileExists($this->settingsProvider->getPathGlobal());
if(SettingsPiwik::isMatomoInstalled()) {
$this->checkConfigFileExists($this->settingsProvider->getPathLocal(), $startInstaller = false);
return;
}
$startInstaller = true;
if(SettingsServer::isTrackerApiRequest()) {
// if Piwik is not installed yet, the piwik.php should do nothing and not return an error
throw new NotYetInstalledException("As Matomo is not installed yet, the Tracking API cannot proceed and will exit without error.");
}
if(Common::isPhpCliMode()) {
// in CLI, do not start/redirect to installer, simply output the exception at the top
$startInstaller = false;
}
// Start the installation when config file not found
$this->checkConfigFileExists($this->settingsProvider->getPathLocal(), $startInstaller);
}
/**
* @param $path
* @param bool $startInstaller
* @throws \Exception
*/
private function checkConfigFileExists($path, $startInstaller = false)
{
if (is_readable($path)) {
return;
}
$general = $this->settingsProvider->getSection('General');
if (isset($general['enable_installer'])
&& !$general['enable_installer']
) {
throw new NotYetInstalledException('Matomo is not set up yet');
}
$message = $this->getSpecificMessageWhetherFileExistsOrNot($path);
$exception = new NotYetInstalledException($message);
if ($startInstaller) {
$this->startInstallation($exception);
} else {
throw $exception;
}
}
/**
* @param $exception
*/
private function startInstallation($exception)
{
/**
* Triggered when the configuration file cannot be found or read, which usually
* means Piwik is not installed yet.
*
* This event can be used to start the installation process or to display a custom error message.
*
* @param \Exception $exception The exception that was thrown by `Config::getInstance()`.
*/
Piwik::postEvent('Config.NoConfigurationFile', array($exception), $pending = true);
}
/**
* @param $path
* @return string
*/
private function getMessageWhenFileExistsButNotReadable($path)
{
$format = " \n<b>» %s </b>";
if(Common::isPhpCliMode()) {
$format = "\n » %s \n";
}
return sprintf($format,
$this->translator->translate('General_ExceptionConfigurationFilePleaseCheckReadableByUser',
array($path, Filechecks::getUser())));
}
/**
* @param $path
* @return string
*/
private function getSpecificMessageWhetherFileExistsOrNot($path)
{
if (!file_exists($path)) {
$message = $this->translator->translate('General_ExceptionConfigurationFileNotFound', array($path));
if (Common::isPhpCliMode()) {
$message .= $this->getMessageWhenFileExistsButNotReadable($path);
}
} else {
$message = $this->translator->translate('General_ExceptionConfigurationFileExistsButNotReadable',
array($path));
$message .= $this->getMessageWhenFileExistsButNotReadable($path);
}
if (Common::isPhpCliMode()) {
$message = "\n" . $message;
}
return $message;
}
}

111
matomo/core/Application/Kernel/GlobalSettingsProvider.php Normal file
View file

@ -0,0 +1,111 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application\Kernel;
use Piwik\Config;
use Piwik\Config\IniFileChain;
/**
* Provides global settings. Global settings are organized in sections where
* each section contains a list of name => value pairs. Setting values can
* be primitive values or arrays of primitive values.
*
* Uses the config.ini.php, common.ini.php and global.ini.php files to provide global settings.
*
* At the moment a singleton instance of this class is used in order to get tests to pass.
*/
class GlobalSettingsProvider
{
/**
* @var IniFileChain
*/
protected $iniFileChain;
/**
* @var string
*/
protected $pathGlobal = null;
/**
* @var string
*/
protected $pathCommon = null;
/**
* @var string
*/
protected $pathLocal = null;
/**
* @param string|null $pathGlobal Path to the global.ini.php file. Or null to use the default.
* @param string|null $pathLocal Path to the config.ini.php file. Or null to use the default.
* @param string|null $pathCommon Path to the common.ini.php file. Or null to use the default.
*/
public function __construct($pathGlobal = null, $pathLocal = null, $pathCommon = null)
{
$this->pathGlobal = $pathGlobal ?: Config::getGlobalConfigPath();
$this->pathCommon = $pathCommon ?: Config::getCommonConfigPath();
$this->pathLocal = $pathLocal ?: Config::getLocalConfigPath();
$this->iniFileChain = new IniFileChain();
$this->reload();
}
public function reload($pathGlobal = null, $pathLocal = null, $pathCommon = null)
{
$this->pathGlobal = $pathGlobal ?: $this->pathGlobal;
$this->pathCommon = $pathCommon ?: $this->pathCommon;
$this->pathLocal = $pathLocal ?: $this->pathLocal;
$this->iniFileChain->reload(array($this->pathGlobal, $this->pathCommon), $this->pathLocal);
}
/**
* Returns a settings section.
*
* @param string $name
* @return array
*/
public function &getSection($name)
{
$section =& $this->iniFileChain->get($name);
return $section;
}
/**
* Sets a settings section.
*
* @param string $name
* @param array $value
*/
public function setSection($name, $value)
{
$this->iniFileChain->set($name, $value);
}
public function getIniFileChain()
{
return $this->iniFileChain;
}
public function getPathGlobal()
{
return $this->pathGlobal;
}
public function getPathLocal()
{
return $this->pathLocal;
}
public function getPathCommon()
{
return $this->pathCommon;
}
}

197
matomo/core/Application/Kernel/PluginList.php Normal file
View file

@ -0,0 +1,197 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Application\Kernel;
use Piwik\Plugin\MetadataLoader;
/**
* Lists the currently activated plugins. Used when setting up Piwik's environment before
* initializing the DI container.
*
* Uses the [Plugins] section in Piwik's INI config to get the activated plugins.
*
* Depends on GlobalSettingsProvider being used.
*
* TODO: parts of Plugin\Manager edit the plugin list; maybe PluginList implementations should be mutable?
*/
class PluginList
{
/**
* @var GlobalSettingsProvider
*/
private $settings;
/**
* Plugins bundled with core package, disabled by default
* @var array
*/
private $corePluginsDisabledByDefault = array(
'DBStats',
'ExamplePlugin',
'ExampleCommand',
'ExampleSettingsPlugin',
'ExampleUI',
'ExampleVisualization',
'ExamplePluginTemplate',
'ExampleTracker',
'ExampleLogTables',
'ExampleReport',
'ExampleAPI',
'ExampleVue',
'MobileAppMeasurable',
'TagManager'
);
// Themes bundled with core package, disabled by default
private $coreThemesDisabledByDefault = array(
'ExampleTheme'
);
public function __construct(GlobalSettingsProvider $settings)
{
$this->settings = $settings;
}
/**
* Returns the list of plugins that should be loaded. Used by the container factory to
* load plugin specific DI overrides.
*
* @return string[]
*/
public function getActivatedPlugins()
{
$section = $this->settings->getSection('Plugins');
$plugins = @$section['Plugins'] ?: array();
return $plugins;
}
/**
* Returns the list of plugins that are bundled with Piwik.
*
* @return string[]
*/
public function getPluginsBundledWithPiwik()
{
$pathGlobal = $this->settings->getPathGlobal();
$section = $this->settings->getIniFileChain()->getFrom($pathGlobal, 'Plugins');
return $section['Plugins'];
}
/**
* Returns the plugins bundled with core package that are disabled by default.
*
* @return string[]
*/
public function getCorePluginsDisabledByDefault()
{
return array_merge($this->corePluginsDisabledByDefault, $this->coreThemesDisabledByDefault);
}
/**
* Sorts an array of plugins in the order they should be loaded. We cannot use DI here as DI is not initialized
* at this stage.
*
* @params string[] $plugins
* @return \string[]
*/
public function sortPlugins(array $plugins)
{
$global = $this->getPluginsBundledWithPiwik();
if (empty($global)) {
return $plugins;
}
// we need to make sure a possibly disabled plugin will be still loaded before any 3rd party plugin
$global = array_merge($global, $this->corePluginsDisabledByDefault);
$global = array_values($global);
$plugins = array_values($plugins);
$defaultPluginsLoadedFirst = array_intersect($global, $plugins);
$otherPluginsToLoadAfterDefaultPlugins = array_diff($plugins, $defaultPluginsLoadedFirst);
// sort by name to have a predictable order for those extra plugins
natcasesort($otherPluginsToLoadAfterDefaultPlugins);
$sorted = array_merge($defaultPluginsLoadedFirst, $otherPluginsToLoadAfterDefaultPlugins);
return $sorted;
}
/**
* Sorts an array of plugins in the order they should be saved in config.ini.php. This basically influences
* the order of the plugin config.php and which config will be loaded first. We want to make sure to require the
* config or a required plugin first before loading the plugin that requires it.
*
* We do not sort using this logic on each request since it is much slower than `sortPlugins()`. The order
* of plugins in config.ini.php is only important for the ContainerFactory. During a regular request it is otherwise
* fine to load the plugins in the order of `sortPlugins()` since we will make sure that required plugins will be
* loaded first in plugin manager.
*
* @param string[] $plugins
* @param array[] $pluginJsonCache For internal testing only
* @return \string[]
*/
public function sortPluginsAndRespectDependencies(array $plugins, $pluginJsonCache = array())
{
$global = $this->getPluginsBundledWithPiwik();
if (empty($global)) {
return $plugins;
}
// we need to make sure a possibly disabled plugin will be still loaded before any 3rd party plugin
$global = array_merge($global, $this->corePluginsDisabledByDefault);
$global = array_values($global);
$plugins = array_values($plugins);
$defaultPluginsLoadedFirst = array_intersect($global, $plugins);
$otherPluginsToLoadAfterDefaultPlugins = array_diff($plugins, $defaultPluginsLoadedFirst);
// we still want to sort alphabetically by default
natcasesort($otherPluginsToLoadAfterDefaultPlugins);
$sorted = array();
foreach ($otherPluginsToLoadAfterDefaultPlugins as $pluginName) {
$sorted = $this->sortRequiredPlugin($pluginName, $pluginJsonCache, $otherPluginsToLoadAfterDefaultPlugins, $sorted);
}
$sorted = array_merge($defaultPluginsLoadedFirst, $sorted);
return $sorted;
}
private function sortRequiredPlugin($pluginName, &$pluginJsonCache, $toBeSorted, $sorted)
{
if (!isset($pluginJsonCache[$pluginName])) {
$loader = new MetadataLoader($pluginName);
$pluginJsonCache[$pluginName] = $loader->loadPluginInfoJson();
}
if (!empty($pluginJsonCache[$pluginName]['require'])) {
$dependencies = $pluginJsonCache[$pluginName]['require'];
foreach ($dependencies as $possiblePluginName => $key) {
if (in_array($possiblePluginName, $toBeSorted, true) && !in_array($possiblePluginName, $sorted, true)) {
$sorted = $this->sortRequiredPlugin($possiblePluginName, $pluginJsonCache, $toBeSorted, $sorted);
}
}
}
if (!in_array($pluginName, $sorted, true)) {
$sorted[] = $pluginName;
}
return $sorted;
}
}

945
matomo/core/Archive.php Normal file
View file

@ -0,0 +1,945 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
use Piwik\Archive\ArchiveQuery;
use Piwik\Archive\ArchiveQueryFactory;
use Piwik\Archive\DataCollection;
use Piwik\Archive\Parameters;
use Piwik\ArchiveProcessor\Rules;
use Piwik\Container\StaticContainer;
use Piwik\DataAccess\ArchiveSelector;
use Piwik\Plugins\CoreAdminHome\API;
/**
* The **Archive** class is used to query cached analytics statistics
* (termed "archive data").
*
* You can use **Archive** instances to get data that was archived for one or more sites,
* for one or more periods and one optional segment.
*
* If archive data is not found, this class will initiate the archiving process. [1](#footnote-1)
*
* **Archive** instances must be created using the {@link build()} factory method;
* they cannot be constructed.
*
* You can search for metrics (such as `nb_visits`) using the {@link getNumeric()} and
* {@link getDataTableFromNumeric()} methods. You can search for
* reports using the {@link getBlob()}, {@link getDataTable()} and {@link getDataTableExpanded()} methods.
*
* If you're creating an API that returns report data, you may want to use the
* {@link createDataTableFromArchive()} helper function.
*
* ### Learn more
*
* Learn more about _archiving_ [here](/guides/all-about-analytics-data).
*
* ### Limitations
*
* - You cannot get data for multiple range periods in a single query.
* - You cannot get data for periods of different types in a single query.
*
* ### Examples
*
* **_Querying metrics for an API method_**
*
* // one site and one period
* $archive = Archive::build($idSite = 1, $period = 'week', $date = '2013-03-08');
* return $archive->getDataTableFromNumeric(array('nb_visits', 'nb_actions'));
*
* // all sites and multiple dates
* $archive = Archive::build($idSite = 'all', $period = 'month', $date = '2013-01-02,2013-03-08');
* return $archive->getDataTableFromNumeric(array('nb_visits', 'nb_actions'));
*
* **_Querying and using metrics immediately_**
*
* // one site and one period
* $archive = Archive::build($idSite = 1, $period = 'week', $date = '2013-03-08');
* $data = $archive->getNumeric(array('nb_visits', 'nb_actions'));
*
* $visits = $data['nb_visits'];
* $actions = $data['nb_actions'];
*
* // ... do something w/ metric data ...
*
* // multiple sites and multiple dates
* $archive = Archive::build($idSite = '1,2,3', $period = 'month', $date = '2013-01-02,2013-03-08');
* $data = $archive->getNumeric('nb_visits');
*
* $janSite1Visits = $data['1']['2013-01-01,2013-01-31']['nb_visits'];
* $febSite1Visits = $data['1']['2013-02-01,2013-02-28']['nb_visits'];
* // ... etc.
*
* **_Querying for reports_**
*
* $archive = Archive::build($idSite = 1, $period = 'week', $date = '2013-03-08');
* $dataTable = $archive->getDataTable('MyPlugin_MyReport');
* // ... manipulate $dataTable ...
* return $dataTable;
*
* **_Querying a report for an API method_**
*
* public function getMyReport($idSite, $period, $date, $segment = false, $expanded = false)
* {
* $dataTable = Archive::createDataTableFromArchive('MyPlugin_MyReport', $idSite, $period, $date, $segment, $expanded);
* return $dataTable;
* }
*
* **_Querying data for multiple range periods_**
*
* // get data for first range
* $archive = Archive::build($idSite = 1, $period = 'range', $date = '2013-03-08,2013-03-12');
* $dataTable = $archive->getDataTableFromNumeric(array('nb_visits', 'nb_actions'));
*
* // get data for second range
* $archive = Archive::build($idSite = 1, $period = 'range', $date = '2013-03-15,2013-03-20');
* $dataTable = $archive->getDataTableFromNumeric(array('nb_visits', 'nb_actions'));
*
* <a name="footnote-1"></a>
* [1]: The archiving process will not be launched if browser archiving is disabled
* and the current request came from a browser.
*
*
* @api
*/
class Archive implements ArchiveQuery
{
const REQUEST_ALL_WEBSITES_FLAG = 'all';
const ARCHIVE_ALL_PLUGINS_FLAG = 'all';
const ID_SUBTABLE_LOAD_ALL_SUBTABLES = 'all';
/**
* List of archive IDs for the site, periods and segment we are querying with.
* Archive IDs are indexed by done flag and period, ie:
*
* array(
* 'done.Referrers' => array(
* '2010-01-01' => 1,
* '2010-01-02' => 2,
* ),
* 'done.VisitsSummary' => array(
* '2010-01-01' => 3,
* '2010-01-02' => 4,
* ),
* )
*
* or,
*
* array(
* 'done.all' => array(
* '2010-01-01' => 1,
* '2010-01-02' => 2
* )
* )
*
* @var array
*/
private $idarchives = [];
/**
* If set to true, the result of all get functions (ie, getNumeric, getBlob, etc.)
* will be indexed by the site ID, even if we're only querying data for one site.
*
* @var bool
*/
private $forceIndexedBySite;
/**
* If set to true, the result of all get functions (ie, getNumeric, getBlob, etc.)
* will be indexed by the period, even if we're only querying data for one period.
*
* @var bool
*/
private $forceIndexedByDate;
/**
* @var Parameters
*/
private $params;
/**
* @var \Matomo\Cache\Cache
*/
private static $cache;
/**
* If true, this Archive instance will not launch the archiving process, even if the current request
* is authorized to.
*
* @var bool
*/
private $forceFetchingWithoutLaunchingArchiving;
/**
* @param Parameters $params
* @param bool $forceIndexedBySite Whether to force index the result of a query by site ID.
* @param bool $forceIndexedByDate Whether to force index the result of a query by period.
*/
public function __construct(
Parameters $params,
$forceIndexedBySite = false,
$forceIndexedByDate = false
) {
$this->params = $params;
$this->forceIndexedBySite = $forceIndexedBySite;
$this->forceIndexedByDate = $forceIndexedByDate;
}
/**
* Returns a new Archive instance that will query archive data for the given set of
* sites and periods, using an optional Segment.
*
* This method uses data that is found in query parameters, so the parameters to this
* function can be string values.
*
* If you want to create an Archive instance with an array of Period instances, use
* {@link Archive::factory()}.
*
* @param string|int|array $idSites A single ID (eg, `'1'`), multiple IDs (eg, `'1,2,3'` or `array(1, 2, 3)`),
* or `'all'`.
* @param string $period 'day', `'week'`, `'month'`, `'year'` or `'range'`
* @param Date|string $strDate 'YYYY-MM-DD', magic keywords (ie, 'today'; {@link Date::factory()}
* or date range (ie, 'YYYY-MM-DD,YYYY-MM-DD').
* @param bool|false|string $segment Segment definition or false if no segment should be used. {@link Piwik\Segment}
* @param bool|false|string $_restrictSitesToLogin Used only when running as a scheduled task.
* @return ArchiveQuery
*/
public static function build($idSites, $period, $strDate, $segment = false, $_restrictSitesToLogin = false)
{
return StaticContainer::get(ArchiveQueryFactory::class)->build(
$idSites,
$period,
$strDate,
$segment,
$_restrictSitesToLogin
);
}
/**
* Returns a new Archive instance that will query archive data for the given set of
* sites and periods, using an optional segment.
*
* This method uses an array of Period instances and a Segment instance, instead of strings
* like {@link build()}.
*
* If you want to create an Archive instance using data found in query parameters,
* use {@link build()}.
*
* @param Segment $segment The segment to use. For no segment, use `new Segment('', $idSites)`.
* @param array $periods An array of Period instances.
* @param array $idSites An array of site IDs (eg, `array(1, 2, 3)`).
* @param bool $idSiteIsAll Whether `'all'` sites are being queried or not. If true, then
* the result of querying functions will be indexed by site, regardless
* of whether `count($idSites) == 1`.
* @param bool $isMultipleDate Whether multiple dates are being queried or not. If true, then
* the result of querying functions will be indexed by period,
* regardless of whether `count($periods) == 1`.
*
* @return ArchiveQuery
*/
public static function factory(
Segment $segment,
array $periods,
array $idSites,
$idSiteIsAll = false,
$isMultipleDate = false
) {
return StaticContainer::get(ArchiveQueryFactory::class)->factory(
$segment,
$periods,
$idSites,
$idSiteIsAll,
$isMultipleDate
);
}
public static function shouldSkipArchiveIfSkippingSegmentArchiveForToday(Site $site, Period $period, Segment $segment)
{
$now = Date::factory('now', $site->getTimezone());
return $period->getLabel() === 'day'
&& !$segment->isEmpty()
&& $period->getDateStart()->toString() === $now->toString();
}
/**
* Queries and returns metric data in an array.
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be indexed by site ID.
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be indexed by period.
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be indexed by site ID first, then period.
*
* @param string|array $names One or more archive names, eg, `'nb_visits'`, `'Referrers_distinctKeywords'`,
* etc.
* @return false|integer|array `false` if there is no data to return, a single numeric value if we're not querying
* for multiple sites/periods, or an array if multiple sites, periods or names are
* queried for.
*/
public function getNumeric($names)
{
$data = $this->get($names, 'numeric');
$resultIndices = $this->getResultIndices();
$result = $data->getIndexedArray($resultIndices);
// if only one metric is returned, just return it as a numeric value
if (
empty($resultIndices)
&& count($result) <= 1
&& (!is_array($names) || count($names) === 1)
) {
$result = (float)reset($result); // convert to float in case $result is empty
}
return $result;
}
/**
* Queries and returns blob records without turning them into DataTables.
*
* Unlike other methods, this returns a DataCollection instance directly. Use it to directly access
* and process blob data.
*
* @param string|string[] $names One or more archive names, eg, `'nb_visits'`, `'Referrers_distinctKeywords'`,
* etc.
* @return DataCollection the queried data.
*/
public function getBlob($names, $idSubtable = null)
{
return $this->get($names, 'blob', $idSubtable);
}
/**
* Queries and returns metric data in a DataTable instance.
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be a DataTable\Map that is indexed by site ID.
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be a {@link DataTable\Map} that is indexed by period.
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be a {@link DataTable\Map} indexed by site ID which contains {@link DataTable\Map} instances that are
* indexed by period.
*
* _Note: Every DataTable instance returned will have at most one row in it. The contents of each
* row will be the requested metrics for the appropriate site and period._
*
* @param string|array $names One or more archive names, eg, 'nb_visits', 'Referrers_distinctKeywords',
* etc.
* @return DataTable|DataTable\Map A DataTable if multiple sites and periods were not requested.
* An appropriately indexed DataTable\Map if otherwise.
*/
public function getDataTableFromNumeric($names)
{
$data = $this->get($names, 'numeric');
return $data->getDataTable($this->getResultIndices());
}
/**
* Similar to {@link getDataTableFromNumeric()} but merges all children on the created DataTable.
*
* This is the same as doing `$this->getDataTableFromNumeric()->mergeChildren()` but this way it is much faster.
*
* @return DataTable|DataTable\Map
*
* @internal Currently only used by MultiSites.getAll plugin. Feel free to remove internal tag if needed somewhere
* else. If no longer needed by MultiSites.getAll please remove this method. If you need this to work in
* a bit different way feel free to refactor as always.
*/
public function getDataTableFromNumericAndMergeChildren($names)
{
$data = $this->get($names, 'numeric');
$resultIndexes = $this->getResultIndices();
return $data->getMergedDataTable($resultIndexes);
}
/**
* Queries and returns one or more reports as DataTable instances.
*
* This method will query blob data that is a serialized array of of {@link DataTable\Row}'s and
* unserialize it.
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be a {@link DataTable\Map} that is indexed by site ID.
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be a DataTable\Map that is indexed by period.
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be a {@link DataTable\Map} indexed by site ID which contains {@link DataTable\Map} instances that are
* indexed by period.
*
* @param string $name The name of the record to get. This method can only query one record at a time.
* @param int|string|null $idSubtable The ID of the subtable to get (if any).
* @return DataTable|DataTable\Map A DataTable if multiple sites and periods were not requested.
* An appropriately indexed {@link DataTable\Map} if otherwise.
*/
public function getDataTable($name, $idSubtable = null)
{
$data = $this->get($name, 'blob', $idSubtable);
return $data->getDataTable($this->getResultIndices());
}
/**
* Queries and returns one report with all of its subtables loaded.
*
* If multiple sites were requested in {@link build()} or {@link factory()} the result will
* be a DataTable\Map that is indexed by site ID.
*
* If multiple periods were requested in {@link build()} or {@link factory()} the result will
* be a DataTable\Map that is indexed by period.
*
* The site ID index is always first, so if multiple sites & periods were requested, the result
* will be a {@link DataTable\Map indexed} by site ID which contains {@link DataTable\Map} instances that are
* indexed by period.
*
* @param string $name The name of the record to get.
* @param int|string|null $idSubtable The ID of the subtable to get (if any). The subtable will be expanded.
* @param int|null $depth The maximum number of subtable levels to load. If null, all levels are loaded.
* For example, if `1` is supplied, then the DataTable returned will have its subtables
* loaded. Those subtables, however, will NOT have their subtables loaded.
* @param bool $addMetadataSubtableId Whether to add the database subtable ID as metadata to each datatable,
* or not.
* @return DataTable|DataTable\Map
*/
public function getDataTableExpanded($name, $idSubtable = null, $depth = null, $addMetadataSubtableId = true)
{
$data = $this->get($name, 'blob', self::ID_SUBTABLE_LOAD_ALL_SUBTABLES);
return $data->getExpandedDataTable($this->getResultIndices(), $idSubtable, $depth, $addMetadataSubtableId);
}
/**
* Returns the list of plugins that archive the given reports.
*
* @param array $archiveNames
* @return array
*/
private function getRequestedPlugins($archiveNames)
{
$result = [];
foreach ($archiveNames as $name) {
$result[] = self::getPluginForReport($name);
}
return array_unique($result);
}
/**
* Returns an object describing the set of sites, the set of periods and the segment
* this Archive will query data for.
*
* @return Parameters
*/
public function getParams()
{
return $this->params;
}
/**
* Helper function that creates an Archive instance and queries for report data using
* query parameter data. API methods can use this method to reduce code redundancy.
*
* @param string $recordName The name of the report to return.
* @param int|string|array $idSite @see {@link build()}
* @param string $period @see {@link build()}
* @param string $date @see {@link build()}
* @param string $segment @see {@link build()}
* @param bool $expanded If true, loads all subtables. See {@link getDataTableExpanded()}
* @param bool $flat If true, loads all subtables and disabled all recursive filters.
* @param int|null|string $idSubtable See {@link getDataTableExpanded()}
* @param int|null $depth See {@link getDataTableExpanded()}
* @return DataTable|DataTable\Map
*/
public static function createDataTableFromArchive($recordName, $idSite, $period, $date, $segment, $expanded = false, $flat = false, $idSubtable = null, $depth = null)
{
Piwik::checkUserHasViewAccess($idSite);
if ($idSubtable === false || $idSubtable === '') {
$idSubtable = null;
}
if (!empty($idSubtable) && (strtolower($idSubtable) !== self::ID_SUBTABLE_LOAD_ALL_SUBTABLES && !is_numeric($idSubtable))) {
throw new \Exception("idSubtable needs to be a number or '" . self::ID_SUBTABLE_LOAD_ALL_SUBTABLES . "', '$idSubtable' given.");
}
if ($flat && !$idSubtable) {
$expanded = true;
}
$archive = Archive::build($idSite, $period, $date, $segment, $_restrictSitesToLogin = false);
if ($expanded) {
$dataTable = $archive->getDataTableExpanded($recordName, $idSubtable, $depth);
} else {
$dataTable = $archive->getDataTable($recordName, $idSubtable);
}
$dataTable->queueFilter('ReplaceSummaryRowLabel');
$dataTable->queueFilter('ReplaceColumnNames');
if ($expanded) {
$dataTable->queueFilterSubtables('ReplaceColumnNames');
}
if ($flat) {
$dataTable->disableRecursiveFilters();
}
return $dataTable;
}
/**
* Queries archive tables for data and returns the result.
* @param array|string $archiveNames
* @param $archiveDataType
* @param null|int $idSubtable
* @return Archive\DataCollection
*/
protected function get($archiveNames, $archiveDataType, $idSubtable = null)
{
if (!is_array($archiveNames)) {
$archiveNames = [$archiveNames];
}
$archiveNames = array_filter($archiveNames);
// apply idSubtable
if (
$idSubtable !== null
&& $idSubtable !== self::ID_SUBTABLE_LOAD_ALL_SUBTABLES
) {
// this is also done in ArchiveSelector. It should be actually only done in ArchiveSelector but DataCollection
// does require to have the subtableId appended. Needs to be changed in refactoring to have it only in one
// place.
$dataNames = [];
foreach ($archiveNames as $name) {
$dataNames[] = ArchiveSelector::appendIdsubtable($name, $idSubtable);
}
} else {
$dataNames = $archiveNames;
}
$result = new Archive\DataCollection(
$dataNames,
$archiveDataType,
$this->params->getIdSites(),
$this->params->getPeriods(),
$this->params->getSegment(),
$defaultRow = null
);
if (empty($dataNames)) {
return $result; // NOTE: not posting Archive.noArchivedData here, because there might be archive data, someone just requested nothing
}
$archiveIds = $this->getArchiveIds($archiveNames);
if (empty($archiveIds)) {
/**
* Triggered when no archive data is found in an API request.
* @ignore
*/
Piwik::postEvent('Archive.noArchivedData');
return $result;
}
$archiveData = ArchiveSelector::getArchiveData($archiveIds, $archiveNames, $archiveDataType, $idSubtable);
$isNumeric = $archiveDataType === 'numeric';
foreach ($archiveData as $row) {
// values are grouped by idsite (site ID), date1-date2 (date range), then name (field name)
$periodStr = $row['date1'] . ',' . $row['date2'];
if ($isNumeric) {
$row['value'] = $this->formatNumericValue($row['value']);
} else {
$result->addMetadata($row['idsite'], $periodStr, DataTable::ARCHIVED_DATE_METADATA_NAME, $row['ts_archived']);
}
$result->set($row['idsite'], $periodStr, $row['name'], $row['value'], [DataTable::ARCHIVED_DATE_METADATA_NAME => $row['ts_archived']]);
}
return $result;
}
/**
* Returns archive IDs for the sites, periods and archive names that are being
* queried. This function will use the idarchive cache if it has the right data,
* query archive tables for IDs w/o launching archiving, or launch archiving and
* get the idarchive from ArchiveProcessor instances.
*
* @param string $archiveNames
* @return array
*/
private function getArchiveIds($archiveNames)
{
$plugins = $this->getRequestedPlugins($archiveNames);
// figure out which archives haven't been processed (if an archive has been processed,
// then we have the archive IDs in $this->idarchives)
$doneFlags = [];
$archiveGroups = [];
foreach (array_merge($plugins, ['all']) as $plugin) {
$doneFlag = $this->getDoneStringForPlugin($plugin, $this->params->getIdSites());
$doneFlags[$doneFlag] = true;
if (!isset($this->idarchives[$doneFlag])) {
$archiveGroup = $this->getArchiveGroupOfPlugin($plugin);
if ($archiveGroup === self::ARCHIVE_ALL_PLUGINS_FLAG) {
$archiveGroup = reset($plugins);
}
$archiveGroups[] = $archiveGroup;
}
$doneFlag = Rules::getDoneFlagArchiveContainsOnePlugin($this->params->getSegment(), $plugin);
$doneFlags[$doneFlag] = true;
}
$globalDoneFlag = Rules::getDoneFlagArchiveContainsAllPlugins($this->params->getSegment());
$doneFlags[$globalDoneFlag] = true;
$archiveGroups = array_unique($archiveGroups);
// cache id archives for plugins we haven't processed yet
if (!empty($archiveGroups)) {
if (
Rules::isArchivingEnabledFor($this->params->getIdSites(), $this->params->getSegment(), $this->getPeriodLabel())
&& !$this->forceFetchingWithoutLaunchingArchiving
) {
$this->cacheArchiveIdsAfterLaunching($archiveGroups, $plugins);
} else {
$this->cacheArchiveIdsWithoutLaunching($plugins);
}
}
$idArchivesByMonth = $this->getIdArchivesByMonth($doneFlags);
return $idArchivesByMonth;
}
/**
* Gets the IDs of the archives we're querying for and stores them in $this->archives.
* This function will launch the archiving process for each period/site/plugin if
* metrics/reports have not been calculated/archived already.
*
* @param array $archiveGroups @see getArchiveGroupOfReport
* @param array $plugins List of plugin names to archive.
*/
private function cacheArchiveIdsAfterLaunching($archiveGroups, $plugins)
{
foreach ($this->params->getPeriods() as $period) {
$twoDaysAfterPeriod = $period->getDateEnd()->addDay(2);
foreach ($this->params->getIdSites() as $idSite) {
$site = new Site($idSite);
if (
Common::getRequestVar('skipArchiveSegmentToday', 0, 'int')
&& self::shouldSkipArchiveIfSkippingSegmentArchiveForToday($site, $period, $this->params->getSegment())
) {
Log::debug("Skipping archive %s for %s as segment today is disabled", $period->getLabel(), $period->getPrettyString());
continue;
}
// if the END of the period is BEFORE the website creation date
// we already know there are no stats for this period
// we add one day to make sure we don't miss the day of the website creation
if ($twoDaysAfterPeriod->isEarlier($site->getCreationDate())) {
Log::debug(
"Archive site %s, %s (%s) skipped, archive is before the website was created.",
$idSite,
$period->getLabel(),
$period->getPrettyString()
);
continue;
}
// Allow for site timezone, local time may have started a new day ahead of UTC
$today = \Piwik\Date::factory('now', $site->getTimezone());
// if the starting date is in the future we know there are no visits
if ($period->getDateStart()->isLater($today)) {
Log::debug(
"Archive site %s, %s (%s) skipped, archive is after today.",
$idSite,
$period->getLabel(),
$period->getPrettyString()
);
continue;
}
$this->prepareArchive($archiveGroups, $site, $period);
}
}
}
/**
* Gets the IDs of the archives we're querying for and stores them in $this->archives.
* This function will not launch the archiving process (and is thus much, much faster
* than cacheArchiveIdsAfterLaunching).
*
* @param array $plugins List of plugin names from which data is being requested.
*/
private function cacheArchiveIdsWithoutLaunching($plugins)
{
$idarchivesByReport = ArchiveSelector::getArchiveIds(
$this->params->getIdSites(),
$this->params->getPeriods(),
$this->params->getSegment(),
$plugins
);
// initialize archive ID cache for each report
foreach ($plugins as $plugin) {
$doneFlag = $this->getDoneStringForPlugin($plugin, $this->params->getIdSites());
$this->initializeArchiveIdCache($doneFlag);
$globalDoneFlag = Rules::getDoneFlagArchiveContainsAllPlugins($this->params->getSegment());
$this->initializeArchiveIdCache($globalDoneFlag);
}
foreach ($idarchivesByReport as $doneFlag => $idarchivesByDate) {
foreach ($idarchivesByDate as $dateRange => $idarchives) {
foreach ($idarchives as $idarchive) {
// idarchives selected can include all plugin archives, plugin specific archives and partial report
// archives. only the latest data in all of these archives will be selected.
$this->idarchives[$doneFlag][$dateRange][] = $idarchive;
}
}
}
}
/**
* Returns the done string flag for a plugin using this instance's segment & periods.
* @param string $plugin
* @return string
*/
private function getDoneStringForPlugin($plugin, $idSites)
{
return Rules::getDoneStringFlagFor(
$idSites,
$this->params->getSegment(),
$this->getPeriodLabel(),
$plugin
);
}
private function getPeriodLabel()
{
$periods = $this->params->getPeriods();
return reset($periods)->getLabel();
}
/**
* Returns an array describing what metadata to use when indexing a query result.
* For use with DataCollection.
*
* @return array
*/
private function getResultIndices()
{
$indices = [];
if (
count($this->params->getIdSites()) > 1
|| $this->forceIndexedBySite
) {
$indices['site'] = 'idSite';
}
if (
count($this->params->getPeriods()) > 1
|| $this->forceIndexedByDate
) {
$indices['period'] = 'date';
}
return $indices;
}
private function formatNumericValue($value)
{
// If there is no dot, we return as is
// Note: this could be an integer bigger than 32 bits
if (strpos($value, '.') === false) {
if ($value === false) {
return 0;
}
return (float)$value;
}
// Round up the value with 2 decimals
// we cast the result as float because returns false when no visitors
return round((float)$value, 2);
}
/**
* Initializes the archive ID cache ($this->idarchives) for a particular 'done' flag.
*
* It is necessary that each archive ID caching function call this method for each
* unique 'done' flag it encounters, since the getArchiveIds function determines
* whether archiving should be launched based on whether $this->idarchives has a
* an entry for a specific 'done' flag.
*
* If this function is not called, then periods with no visits will not add
* entries to the cache. If the archive is used again, SQL will be executed to
* try and find the archive IDs even though we know there are none.
*
* @param string $doneFlag
*/
private function initializeArchiveIdCache($doneFlag)
{
if (!isset($this->idarchives[$doneFlag])) {
$this->idarchives[$doneFlag] = [];
}
}
/**
* Returns the archiving group identifier given a plugin.
*
* More than one plugin can be called at once when archiving. In such a case
* we don't want to launch archiving three times for three plugins if doing
* it once is enough, so getArchiveIds makes sure to get the archive group of
* all reports.
*
* If the period isn't a range, then all plugins' archiving code is executed.
* If the period is a range, then archiving code is executed individually for
* each plugin.
*/
private function getArchiveGroupOfPlugin($plugin)
{
$periods = $this->params->getPeriods();
$periodLabel = reset($periods)->getLabel();
if (Rules::shouldProcessReportsAllPlugins($this->params->getIdSites(), $this->params->getSegment(), $periodLabel)) {
return self::ARCHIVE_ALL_PLUGINS_FLAG;
}
return $plugin;
}
/**
* Returns the name of the plugin that archives a given report.
*
* @param string $report Archive data name, eg, `'nb_visits'`, `'DevicesDetection_...'`, etc.
* @return string Plugin name.
* @throws \Exception If a plugin cannot be found or if the plugin for the report isn't
* activated.
*/
public static function getPluginForReport($report)
{
if (in_array($report, Metrics::getVisitsMetricNames())) {
// Core metrics are always processed in Core, for the requested date/period/segment
$report = 'VisitsSummary_CoreMetrics';
} elseif (strpos($report, 'Goal_') === 0) {
// Goal_* metrics are processed by the Goals plugin (HACK)
$report = 'Goals_Metrics';
} elseif (
strrpos($report, '_returning') === strlen($report) - strlen('_returning') ||
strrpos($report, '_new') === strlen($report) - strlen('_new')
) { // HACK
$report = 'VisitFrequency_Metrics';
}
$plugin = substr($report, 0, strpos($report, '_'));
if (
empty($plugin)
|| !\Piwik\Plugin\Manager::getInstance()->isPluginActivated($plugin)
) {
throw new \Exception("Error: The report '$report' was requested but it is not available at this stage."
. " (Plugin '$plugin' is not activated.)");
}
return $plugin;
}
/**
* @param $archiveGroups
* @param $site
* @param $period
*/
private function prepareArchive(array $archiveGroups, Site $site, Period $period)
{
$coreAdminHomeApi = API::getInstance();
$requestedReport = null;
if (SettingsServer::isArchivePhpTriggered()) {
$requestedReport = Common::getRequestVar('requestedReport', '', 'string');
}
$periodString = $period->getRangeString();
$periodDateStr = $period->getLabel() == 'range' ? $periodString : $period->getDateStart()->toString();
$idSites = [$site->getId()];
// process for each plugin as well
foreach ($archiveGroups as $plugin) {
$doneFlag = $this->getDoneStringForPlugin($plugin, $idSites);
$this->initializeArchiveIdCache($doneFlag);
$prepareResult = $coreAdminHomeApi->archiveReports(
$site->getId(),
$period->getLabel(),
$periodDateStr,
$this->params->getSegment()->getOriginalString(),
$plugin,
$requestedReport
);
if (
!empty($prepareResult)
&& !empty($prepareResult['idarchives'])
) {
foreach ($prepareResult['idarchives'] as $idArchive) {
$this->idarchives[$doneFlag][$periodString][] = $idArchive;
}
}
}
}
private function getIdArchivesByMonth($doneFlags)
{
// order idarchives by the table month they belong to
$idArchivesByMonth = [];
foreach (array_keys($doneFlags) as $doneFlag) {
if (empty($this->idarchives[$doneFlag])) {
continue;
}
foreach ($this->idarchives[$doneFlag] as $dateRange => $idarchives) {
foreach ($idarchives as $id) {
$idArchivesByMonth[$dateRange][] = $id;
}
}
}
return $idArchivesByMonth;
}
/**
* @internal
*/
public static function clearStaticCache()
{
self::$cache = null;
}
public function forceFetchingWithoutLaunchingArchiving()
{
$this->forceFetchingWithoutLaunchingArchiving = true;
}
}

797
matomo/core/Archive/ArchiveInvalidator.php Normal file
View file

@ -0,0 +1,797 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archive;
use Piwik\Archive\ArchiveInvalidator\InvalidationResult;
use Piwik\ArchiveProcessor\Rules;
use Piwik\Common;
use Piwik\Container\StaticContainer;
use Piwik\CronArchive\ReArchiveList;
use Piwik\CronArchive\SegmentArchiving;
use Piwik\DataAccess\ArchiveTableCreator;
use Piwik\DataAccess\Model;
use Piwik\Date;
use Piwik\Db;
use Piwik\Option;
use Piwik\Period;
use Piwik\Piwik;
use Piwik\Plugin\Manager;
use Piwik\Plugins\CoreAdminHome\Tasks\ArchivesToPurgeDistributedList;
use Piwik\Plugins\PrivacyManager\PrivacyManager;
use Piwik\Segment;
use Piwik\SettingsServer;
use Piwik\Site;
use Piwik\Tracker\Cache;
use Psr\Log\LoggerInterface;
/**
* Service that can be used to invalidate archives or add archive references to a list so they will
* be invalidated later.
*
* Archives are put in an "invalidated" state by setting the done flag to `ArchiveWriter::DONE_INVALIDATED`.
* This class also adds the archive's associated site to the a distributed list and adding the archive's year month to another
* distributed list.
*
* CronArchive will reprocess the archive data for all sites in the first list, and a scheduled task
* will purge the old, invalidated data in archive tables identified by the second list.
*
* Until CronArchive, or browser triggered archiving, re-processes data for an invalidated archive, the invalidated
* archive data will still be displayed in the UI and API.
*
* ### Deferred Invalidation
*
* Invalidating archives means running queries on one or more archive tables. In some situations, like during
* tracking, this is not desired. In such cases, archive references can be added to a list via the
* rememberToInvalidateArchivedReportsLater method, which will add the reference to a distributed list
*
* Later, during Piwik's normal execution, the list will be read and every archive it references will
* be invalidated.
*/
class ArchiveInvalidator
{
const TRACKER_CACHE_KEY = 'ArchiveInvalidator.rememberToInvalidate';
const INVALIDATION_STATUS_QUEUED = 0;
const INVALIDATION_STATUS_IN_PROGRESS = 1;
private $rememberArchivedReportIdStart = 'report_to_invalidate_';
/**
* @var Model
*/
private $model;
/**
* @var SegmentArchiving
*/
private $segmentArchiving;
/**
* @var LoggerInterface
*/
private $logger;
/**
* @var int[]
*/
private $allIdSitesCache;
public function __construct(Model $model, LoggerInterface $logger)
{
$this->model = $model;
$this->segmentArchiving = null;
$this->logger = $logger;
}
public function getAllRememberToInvalidateArchivedReportsLater()
{
// we do not really have to get the value first. we could simply always try to call set() and it would update or
// insert the record if needed but we do not want to lock the table (especially since there are still some
// MyISAM installations)
$values = Option::getLike('%' . str_replace('_', '\_', $this->rememberArchivedReportIdStart) . '%');
$all = [];
foreach ($values as $name => $value) {
$suffix = substr($name, strpos($name, $this->rememberArchivedReportIdStart));
$suffix = str_replace($this->rememberArchivedReportIdStart, '', $suffix);
list($idSite, $dateStr) = explode('_', $suffix);
$all[$idSite][$dateStr] = $value;
}
return $all;
}
public function rememberToInvalidateArchivedReportsLater($idSite, Date $date)
{
if (SettingsServer::isTrackerApiRequest()) {
$value = $this->getRememberedArchivedReportsOptionFromTracker($idSite, $date->toString());
} else {
// To support multiple transactions at once, look for any other process to have set (and committed)
// this report to be invalidated.
$key = $this->buildRememberArchivedReportIdForSiteAndDate($idSite, $date->toString());
// we do not really have to get the value first. we could simply always try to call set() and it would update or
// insert the record if needed but we do not want to lock the table (especially since there are still some
// MyISAM installations)
$value = Option::getLike('%' . str_replace('_', '\_', $key) . '%');
}
// getLike() returns an empty array rather than 'false'
if (empty($value)) {
// In order to support multiple concurrent transactions, add our pid to the end of the key so that it will just insert
// rather than waiting on some other process to commit before proceeding.The issue is that with out this, more than
// one process is trying to add the exact same value to the table, which causes contention. With the pid suffixed to
// the value, each process can successfully enter its own row in the table. The net result will be the same. We could
// always just set this, but it would result in a lot of rows in the options table.. more than needed. With this
// change you'll have at most N rows per date/site, where N is the number of parallel requests on this same idsite/date
// that happen to run in overlapping transactions.
$mykey = $this->buildRememberArchivedReportIdProcessSafe($idSite, $date->toString());
Option::set($mykey, '1');
Cache::clearCacheGeneral();
return $mykey;
}
}
private function getRememberedArchivedReportsOptionFromTracker($idSite, $dateStr)
{
$cacheKey = self::TRACKER_CACHE_KEY;
$generalCache = Cache::getCacheGeneral();
if (empty($generalCache[$cacheKey][$idSite][$dateStr])) {
return [];
}
return $generalCache[$cacheKey][$idSite][$dateStr];
}
public function getRememberedArchivedReportsThatShouldBeInvalidated()
{
$reports = Option::getLike('%' . str_replace('_', '\_', $this->rememberArchivedReportIdStart) . '%\_%');
$sitesPerDay = array();
foreach ($reports as $report => $value) {
$report = substr($report, strpos($report, $this->rememberArchivedReportIdStart));
$report = str_replace($this->rememberArchivedReportIdStart, '', $report);
$report = explode('_', $report);
$siteId = (int) $report[0];
$date = $report[1];
if (empty($siteId)) {
continue;
}
if (empty($sitesPerDay[$date])) {
$sitesPerDay[$date] = array();
}
$sitesPerDay[$date][] = $siteId;
}
return $sitesPerDay;
}
private function buildRememberArchivedReportIdForSite($idSite)
{
return $this->rememberArchivedReportIdStart . (int) $idSite;
}
private function buildRememberArchivedReportIdForSiteAndDate($idSite, $date)
{
$id = $this->buildRememberArchivedReportIdForSite($idSite);
$id .= '_' . trim($date);
return $id;
}
// This version is multi process safe on the insert of a new date to invalidate.
private function buildRememberArchivedReportIdProcessSafe($idSite, $date)
{
$id = Common::getRandomString(4, 'abcdefghijklmnoprstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ') . '_';
$id .= $this->buildRememberArchivedReportIdForSiteAndDate($idSite, $date);
$id .= '_' . Common::getProcessId();
return $id;
}
public function forgetRememberedArchivedReportsToInvalidateForSite($idSite)
{
$id = $this->buildRememberArchivedReportIdForSite($idSite) . '_';
$hasDeletedSomething = $this->deleteOptionLike($id);
if ($hasDeletedSomething) {
Cache::clearCacheGeneral();
}
}
/**
* @internal
* After calling this method, make sure to call Cache::clearCacheGeneral(); For performance reasons we don't call
* this here immediately in case there are multiple invalidations.
*/
public function forgetRememberedArchivedReportsToInvalidate($idSite, Date $date)
{
$id = $this->buildRememberArchivedReportIdForSiteAndDate($idSite, $date->toString());
// The process pid is added to the end of the entry in order to support multiple concurrent transactions.
// So this must be a deleteLike call to get all the entries, where there used to only be one.
return $this->deleteOptionLike($id);
}
/**
* @param $id
* @return bool true if a record was deleted, false otherwise.
* @throws \Zend_Db_Statement_Exception
*/
private function deleteOptionLike($id)
{
// we're not using deleteLike since it maybe could cause deadlocks see https://github.com/matomo-org/matomo/issues/15545
// we want to reduce number of rows scanned and only delete specific primary key
$keys = Option::getLike('%' . str_replace('_', '\_', $id) . '%');
if (empty($keys)) {
return false;
}
$keys = array_keys($keys);
$placeholders = Common::getSqlStringFieldsArray($keys);
$table = Common::prefixTable('option');
$db = Db::query('DELETE FROM `' . $table . '` WHERE `option_name` IN (' . $placeholders . ')', $keys);
return (bool) $db->rowCount();
}
/**
* @param $idSites int[]
* @param $dates Date[]|string[]
* @param $period string
* @param $segment Segment
* @param bool $cascadeDown
* @param bool $forceInvalidateNonexistentRanges set true to force inserting rows for ranges in archive_invalidations
* @param string $name null to make sure every plugin is archived when this invalidation is processed by core:archive,
* or a plugin name to only archive the specific plugin.
* @param bool $ignorePurgeLogDataDate
* @return InvalidationResult
* @throws \Exception
*/
public function markArchivesAsInvalidated(array $idSites, array $dates, $period, Segment $segment = null, $cascadeDown = false,
$forceInvalidateNonexistentRanges = false, $name = null, $ignorePurgeLogDataDate = false)
{
$plugin = null;
if ($name && strpos($name, '.') !== false) {
list($plugin) = explode('.', $name);
}
if ($plugin
&& !Manager::getInstance()->isPluginActivated($plugin)
) {
throw new \Exception("Plugin is not activated: '$plugin'");
}
$invalidationInfo = new InvalidationResult();
// quick fix for #15086, if we're only invalidating today's date for a site, don't add the site to the list of sites
// to reprocess.
$hasMoreThanJustToday = [];
foreach ($idSites as $idSite) {
$hasMoreThanJustToday[$idSite] = true;
$tz = Site::getTimezoneFor($idSite);
if (($period == 'day' || $period === false)
&& count($dates) == 1
&& ((string)$dates[0]) == ((string)Date::factoryInTimezone('today', $tz))
) {
// date is for today
$hasMoreThanJustToday[$idSite] = false;
}
}
/**
* Triggered when a Matomo user requested the invalidation of some reporting archives. Using this event, plugin
* developers can automatically invalidate another site, when a site is being invalidated. A plugin may even
* remove an idSite from the list of sites that should be invalidated to prevent it from ever being
* invalidated.
*
* **Example**
*
* public function getIdSitesToMarkArchivesAsInvalidates(&$idSites)
* {
* if (in_array(1, $idSites)) {
* $idSites[] = 5; // when idSite 1 is being invalidated, also invalidate idSite 5
* }
* }
*
* @param array &$idSites An array containing a list of site IDs which are requested to be invalidated.
*/
Piwik::postEvent('Archiving.getIdSitesToMarkArchivesAsInvalidated', array(&$idSites));
// we trigger above event on purpose here and it is good that the segment was created like
// `new Segment($segmentString, $idSites)` because when a user adds a site via this event, the added idSite
// might not have this segment meaning we avoid a possible error. For the workflow to work, any added or removed
// idSite does not need to be added to $segment.
$datesToInvalidate = $this->removeDatesThatHaveBeenPurged($dates, $period, $invalidationInfo, $ignorePurgeLogDataDate);
$allPeriodsToInvalidate = $this->getAllPeriodsByYearMonth($period, $datesToInvalidate, $cascadeDown);
$this->markArchivesInvalidated($idSites, $allPeriodsToInvalidate, $segment, $period != 'range', $forceInvalidateNonexistentRanges, $name);
$isInvalidatingDays = $period == 'day' || $cascadeDown || empty($period);
$isNotInvalidatingSegment = empty($segment) || empty($segment->getString());
if ($isInvalidatingDays
&& $isNotInvalidatingSegment
) {
$hasDeletedAny = false;
foreach ($idSites as $idSite) {
foreach ($dates as $date) {
if (is_string($date)) {
$date = Date::factory($date);
}
$hasDeletedAny = $this->forgetRememberedArchivedReportsToInvalidate($idSite, $date) || $hasDeletedAny;
}
}
if ($hasDeletedAny) {
Cache::clearCacheGeneral();
}
}
return $invalidationInfo;
}
private function getAllPeriodsByYearMonth($periodOrAll, $dates, $cascadeDown, &$result = [])
{
$periods = $periodOrAll ? [$periodOrAll] : ['day'];
foreach ($periods as $period) {
foreach ($dates as $date) {
$periodObj = $this->makePeriod($date, $period);
$result[$this->getYearMonth($periodObj)][$this->getUniquePeriodId($periodObj)] = $periodObj;
// cascade down
if ($cascadeDown
&& $period != 'range'
) {
$this->addChildPeriodsByYearMonth($result, $periodObj);
}
// cascade up
// if the period spans multiple years or months, it won't be used when aggregating parent periods, so
// we can avoid invalidating it
if ($this->shouldPropagateUp($periodObj)
&& $period != 'range'
) {
$this->addParentPeriodsByYearMonth($result, $periodObj);
}
}
}
return $result;
}
private function shouldPropagateUp(Period $periodObj)
{
return $periodObj->getDateStart()->toString('Y') == $periodObj->getDateEnd()->toString('Y')
&& $periodObj->getDateStart()->toString('m') == $periodObj->getDateEnd()->toString('m');
}
private function addChildPeriodsByYearMonth(&$result, Period $period)
{
if ($period->getLabel() == 'range') {
return;
} else if ($period->getLabel() == 'day'
&& $this->shouldPropagateUp($period)
) {
$this->addParentPeriodsByYearMonth($result, $period);
return;
}
foreach ($period->getSubperiods() as $subperiod) {
$result[$this->getYearMonth($subperiod)][$this->getUniquePeriodId($subperiod)] = $subperiod;
$this->addChildPeriodsByYearMonth($result, $subperiod);
}
}
private function addParentPeriodsByYearMonth(&$result, Period $period, Date $originalDate = null)
{
if ($period->getLabel() == 'year'
|| $period->getLabel() == 'range'
|| !Period\Factory::isPeriodEnabledForAPI($period->getParentPeriodLabel())
) {
return;
}
$originalDate = $originalDate ?? $period->getDateStart();
$parentPeriod = Period\Factory::build($period->getParentPeriodLabel(), $originalDate);
$result[$this->getYearMonth($parentPeriod)][$this->getUniquePeriodId($parentPeriod)] = $parentPeriod;
$this->addParentPeriodsByYearMonth($result, $parentPeriod, $originalDate);
}
/**
* @param $idSites int[]
* @param $dates Date[]
* @param $period string
* @param $segment Segment
* @param bool $cascadeDown
* @return InvalidationResult
* @throws \Exception
*/
public function markArchivesOverlappingRangeAsInvalidated(array $idSites, array $dates, Segment $segment = null)
{
$invalidationInfo = new InvalidationResult();
$ranges = array();
foreach ($dates as $dateRange) {
$ranges[] = Period\Factory::build('range', $dateRange[0] . ',' . $dateRange[1]);
}
$invalidatedMonths = array();
$archiveNumericTables = ArchiveTableCreator::getTablesArchivesInstalled($type = ArchiveTableCreator::NUMERIC_TABLE);
foreach ($archiveNumericTables as $table) {
$tableDate = ArchiveTableCreator::getDateFromTableName($table);
$rowsAffected = $this->model->updateArchiveAsInvalidated($table, $idSites, $ranges, $segment);
if ($rowsAffected > 0) {
$invalidatedMonths[] = $tableDate;
}
}
foreach ($idSites as $idSite) {
foreach ($dates as $dateRange) {
$this->forgetRememberedArchivedReportsToInvalidate($idSite, $dateRange[0]);
$invalidationInfo->processedDates[] = $dateRange[0];
}
}
Cache::clearCacheGeneral();
return $invalidationInfo;
}
/**
* Schedule rearchiving of reports for a single plugin or single report for N months in the past. The next time
* core:archive is run, they will be processed.
*
* @param int[]|string $idSites A list of idSites or 'all'
* @param string $plugin
* @param string|null $report
* @param Date|null $startDate
* @throws \Exception
* @api
*/
public function reArchiveReport($idSites, string $plugin = null, string $report = null, Date $startDate = null, Segment $segment = null)
{
$date2 = Date::today();
$earliestDateToRearchive = Piwik::getEarliestDateToRearchive();
if (empty($startDate)) {
if (empty($earliestDateToRearchive)) {
return null; // INI setting set to 0 months so no rearchiving
}
$startDate = $earliestDateToRearchive;
} else if (!empty($earliestDateToRearchive)) {
// don't allow archiving further back than the rearchive_reports_in_past_last_n_months date allows
$startDate = $startDate->isEarlier($earliestDateToRearchive) ? $earliestDateToRearchive : $startDate;
}
if ($idSites === 'all') {
$idSites = $this->getAllSitesId();
}
$dates = [];
$date = $startDate;
while ($date->isEarlier($date2)) {
$dates[] = $date;
$date = $date->addDay(1);
}
if (empty($dates)) {
return;
}
$name = $plugin;
if (!empty($report)) {
$name .= '.' . $report;
}
$this->markArchivesAsInvalidated($idSites, $dates, 'day', $segment, $cascadeDown = false, $forceInvalidateRanges = false, $name);
if (empty($segment)
&& Rules::shouldProcessSegmentsWhenReArchivingReports()
) {
foreach ($idSites as $idSite) {
foreach (Rules::getSegmentsToProcess([$idSite]) as $segment) {
$this->markArchivesAsInvalidated($idSites, $dates, 'day', new Segment($segment, [$idSite]),
$cascadeDown = false, $forceInvalidateRanges = false, $name);
}
}
}
}
/**
* Remove invalidations for a specific report or all invalidations for a specific plugin. If your plugin supports
* archiving data in the past, you may want to call this method to remove any pending invalidations if, for example,
* your plugin is deactivated or a report deleted.
*
* @param int|int[] $idSite one or more site IDs or 'all' for all site IDs
* @param string $string
* @param string|null $report
*/
public function removeInvalidations($idSite, $plugin, $report = null)
{
if (empty($report)) {
$this->model->removeInvalidationsLike($idSite, $plugin);
} else {
$this->model->removeInvalidations($idSite, $plugin, $report);
}
}
/**
* Schedules a re-archiving reports without propagating exceptions. This is scheduled
* since adding invalidations can take a long time and delay UI response times.
*
* @param int|int[]|'all' $idSites
* @param string|int $pluginName
* @param string|null $report
* @param Date|null $startDate
*/
public function scheduleReArchiving($idSites, string $pluginName = null, $report = null, Date $startDate = null,
Segment $segment = null)
{
if (!empty($report)) {
$this->removeInvalidationsSafely($idSites, $pluginName, $report);
}
try {
$reArchiveList = new ReArchiveList($this->logger);
$reArchiveList->add(json_encode([
'idSites' => $idSites,
'pluginName' => $pluginName,
'report' => $report,
'startDate' => $startDate ? $startDate->getTimestamp() : null,
'segment' => $segment ? $segment->getOriginalString() : null,
]));
} catch (\Throwable $ex) {
$this->logger->info("Failed to schedule rearchiving of past reports for $pluginName plugin.");
}
}
/**
* Applies the queued archiving rearchiving entries.
*/
public function applyScheduledReArchiving()
{
$reArchiveList = new ReArchiveList($this->logger);
$items = $reArchiveList->getAll();
foreach ($items as $item) {
try {
$entry = @json_decode($item, true);
if (empty($entry)) {
continue;
}
$idSites = Site::getIdSitesFromIdSitesString($entry['idSites']);
$this->reArchiveReport(
$idSites,
$entry['pluginName'],
$entry['report'],
!empty($entry['startDate']) ? Date::factory((int) $entry['startDate']) : null,
!empty($entry['segment']) ? new Segment($entry['segment'], $idSites) : null
);
} catch (\Throwable $ex) {
$this->logger->info("Failed to create invalidations for report re-archiving (idSites = {idSites}, pluginName = {pluginName}, report = {report}, startDate = {startDateTs}): {ex}", [
'idSites' => json_encode($entry['idSites']),
'pluginName' => $entry['pluginName'],
'report' => $entry['report'],
'startDateTs' => $entry['startDate'],
'ex' => $ex,
]);
} finally {
$reArchiveList->remove([$item]);
}
}
}
/**
* Calls removeInvalidations() without propagating exceptions.
*
* @param int|int[]|'all' $idSites
* @param string $pluginName
* @param string|null $report
*/
public function removeInvalidationsSafely($idSites, $pluginName, $report = null)
{
try {
$this->removeInvalidations($idSites, $pluginName, $report);
$this->removeInvalidationsFromDistributedList($idSites, $pluginName, $report);
} catch (\Throwable $ex) {
$logger = StaticContainer::get(LoggerInterface::class);
$logger->debug("Failed to remove invalidations the for $pluginName plugin.");
}
}
public function removeInvalidationsFromDistributedList($idSites, $pluginName = null, $report = null)
{
$list = new ReArchiveList();
$entries = $list->getAll();
if ($idSites === 'all') {
$idSites = $this->getAllSitesId();
}
foreach ($entries as $index => $entry) {
$entry = @json_decode($entry, true);
if (empty($entry)) {
unset($entries[$index]);
continue;
}
$entryPluginName = $entry['pluginName'];
if (!empty($pluginName)
&& $pluginName != $entryPluginName
) {
continue;
}
$entryReport = $entry['report'];
if (!empty($pluginName)
&& !empty($report)
&& $report != $entryReport
) {
continue;
}
$sitesInEntry = $entry['idSites'];
if ($sitesInEntry === 'all') {
$sitesInEntry = $this->getAllSitesId();
}
$diffSites = array_diff($sitesInEntry, $idSites);
if (empty($diffSites)) {
unset($entries[$index]);
continue;
}
$entry['idSites'] = $diffSites;
$entries[$index] = json_encode($entry);
}
$list->setAll(array_values($entries));
}
/**
* @param int[] $idSites
* @param string[][][] $dates
* @throws \Exception
*/
private function markArchivesInvalidated($idSites, $dates, Segment $segment = null, $removeRanges = false,
$forceInvalidateNonexistentRanges = false, $name = null)
{
$idSites = array_map('intval', $idSites);
$yearMonths = [];
foreach ($dates as $tableDate => $datesForTable) {
$tableDateObj = Date::factory($tableDate);
$table = ArchiveTableCreator::getNumericTable($tableDateObj);
$yearMonths[] = $tableDateObj->toString('Y_m');
$this->model->updateArchiveAsInvalidated($table, $idSites, $datesForTable, $segment, $forceInvalidateNonexistentRanges, $name);
if ($removeRanges) {
$this->model->updateRangeArchiveAsInvalidated($table, $idSites, $datesForTable, $segment);
}
}
$this->markInvalidatedArchivesForReprocessAndPurge($yearMonths);
}
/**
* @param Date[] $dates
* @param InvalidationResult $invalidationInfo
* @return \Piwik\Date[]
*/
private function removeDatesThatHaveBeenPurged($dates, $period, InvalidationResult $invalidationInfo, $ignorePurgeLogDataDate)
{
$this->findOlderDateWithLogs($invalidationInfo);
$result = array();
foreach ($dates as $date) {
$periodObj = $this->makePeriod($date, $period ?: 'day');
// we should only delete reports for dates that are more recent than N days
if ($invalidationInfo->minimumDateWithLogs
&& !$ignorePurgeLogDataDate
&& ($periodObj->getDateEnd()->isEarlier($invalidationInfo->minimumDateWithLogs)
|| $periodObj->getDateStart()->isEarlier($invalidationInfo->minimumDateWithLogs))
) {
$invalidationInfo->warningDates[] = $date;
continue;
}
$result[] = $date;
$invalidationInfo->processedDates[] = $date;
}
return $result;
}
private function findOlderDateWithLogs(InvalidationResult $info)
{
// If using the feature "Delete logs older than N days"...
$purgeDataSettings = PrivacyManager::getPurgeDataSettings();
$logsDeletedWhenOlderThanDays = (int)$purgeDataSettings['delete_logs_older_than'];
$logsDeleteEnabled = $purgeDataSettings['delete_logs_enable'];
if ($logsDeleteEnabled
&& $logsDeletedWhenOlderThanDays
) {
$info->minimumDateWithLogs = Date::factory('today')->subDay($logsDeletedWhenOlderThanDays);
}
}
/**
* @param array $idSites
* @param array $yearMonths
*/
private function markInvalidatedArchivesForReprocessAndPurge($yearMonths)
{
$archivesToPurge = new ArchivesToPurgeDistributedList();
$archivesToPurge->add($yearMonths);
}
private function getYearMonth(Period $period)
{
return $period->getDateStart()->toString('Y-m-01');
}
private function getUniquePeriodId(Period $period)
{
return $period->getId() . '.' . $period->getRangeString();
}
private function makePeriod($date, $period)
{
if ($period === 'range'
&& strpos($date, ',') === false
) {
$date = $date . ',' . $date;
return new Period\Range('range', $date);
} else {
return Period\Factory::build($period, $date);
}
}
private function getSegmentArchiving()
{
if (empty($this->segmentArchiving)) {
$this->segmentArchiving = new SegmentArchiving(StaticContainer::get('ini.General.process_new_segments_from'));
}
return $this->segmentArchiving;
}
private function getAllSitesId()
{
if (isset($this->allIdSitesCache)) {
return $this->allIdSitesCache;
}
$model = new \Piwik\Plugins\SitesManager\Model();
$this->allIdSitesCache = $model->getSitesId();
return $this->allIdSitesCache;
}
}

57
matomo/core/Archive/ArchiveInvalidator/InvalidationResult.php Normal file
View file

@ -0,0 +1,57 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Archive\ArchiveInvalidator;
use Piwik\Date;
/**
* Information about the result of an archive invalidation operation.
*/
class InvalidationResult
{
/**
* Dates that couldn't be invalidated because they are earlier than the configured log
* deletion limit.
*
* @var array
*/
public $warningDates = [];
/**
* Dates that were successfully invalidated.
*
* @var array
*/
public $processedDates = [];
/**
* The day of the oldest log entry.
*
* @var Date|bool
*/
public $minimumDateWithLogs = false;
/**
* @return string[]
*/
public function makeOutputLogs(): array
{
$output = [];
if ($this->warningDates) {
$output[] = 'Warning: the following Dates have not been invalidated, because they are earlier than your Log Deletion limit: ' .
implode(", ", $this->warningDates) .
"\n The last day with logs is " . $this->minimumDateWithLogs . ". " .
"\n Please disable 'Delete old Logs' or set it to a higher deletion threshold (eg. 180 days or 365 years).'.";
}
$output[] = "Success. The following dates were invalidated successfully: " . implode(", ", $this->processedDates);
return $output;
}
}

330
matomo/core/Archive/ArchivePurger.php Normal file
View file

@ -0,0 +1,330 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archive;
use Piwik\ArchiveProcessor\Rules;
use Piwik\Config;
use Piwik\Container\StaticContainer;
use Piwik\DataAccess\ArchiveTableCreator;
use Piwik\DataAccess\Model;
use Piwik\Date;
use Piwik\Piwik;
use Psr\Log\LoggerInterface;
use Psr\Log\LogLevel;
/**
* Service that purges temporary, error-ed, invalid and custom range archives from archive tables.
*
* Temporary archives are purged if they were archived before a specific time. The time is dependent
* on whether browser triggered archiving is enabled or not.
*
* Error-ed archives are purged w/o constraint.
*
* Invalid archives are purged if a new, valid, archive exists w/ the same site, date, period combination.
* Archives are marked as invalid via Piwik\Archive\ArchiveInvalidator.
*/
class ArchivePurger
{
/**
* @var Model
*/
private $model;
/**
* Date threshold for purging custom range archives. Archives that are older than this date
* are purged unconditionally from the requested archive table.
*
* @var Date
*/
private $purgeCustomRangesOlderThan;
/**
* Date to use for 'yesterday'. Exists so tests can override this value.
*
* @var Date
*/
private $yesterday;
/**
* Date to use for 'today'. Exists so tests can override this value.
*
* @var $today
*/
private $today;
/**
* Date to use for 'now'. Exists so tests can override this value.
*
* @var int
*/
private $now;
/**
* @var LoggerInterface
*/
private $logger;
public function __construct(Model $model = null, Date $purgeCustomRangesOlderThan = null, LoggerInterface $logger = null)
{
$this->model = $model ?: new Model();
$this->purgeCustomRangesOlderThan = $purgeCustomRangesOlderThan ?: self::getDefaultCustomRangeToPurgeAgeThreshold();
$this->yesterday = Date::factory('yesterday');
$this->today = Date::factory('today');
$this->now = time();
$this->logger = $logger ?: StaticContainer::get('Psr\Log\LoggerInterface');
}
/**
* Purge all invalidate archives for whom there are newer, valid archives from the archive
* table that stores data for `$date`.
*
* @param Date $date The date identifying the archive table.
* @return int The total number of archive rows deleted (from both the blog & numeric tables).
*/
public function purgeInvalidatedArchivesFrom(Date $date)
{
$numericTable = ArchiveTableCreator::getNumericTable($date);
$archiveIds = $this->model->getInvalidatedArchiveIdsSafeToDelete($numericTable);
if (empty($archiveIds)) {
$this->logger->debug("No invalidated archives found in {table} with newer, valid archives.", array('table' => $numericTable));
return 0;
}
$emptyIdArchives = $this->model->getPlaceholderArchiveIds($numericTable);
$archiveIds = array_merge($archiveIds, $emptyIdArchives);
$this->logger->info("Found {countArchiveIds} invalidated archives safe to delete in {table}.", array(
'table' => $numericTable, 'countArchiveIds' => count($archiveIds)
));
$deletedRowCount = $this->deleteArchiveIds($date, $archiveIds);
$this->logger->debug("Deleted {count} rows in {table} and its associated blob table.", array(
'table' => $numericTable, 'count' => $deletedRowCount
));
return $deletedRowCount;
}
/**
* Removes the outdated archives for the given month.
* (meaning they are marked with a done flag of ArchiveWriter::DONE_OK_TEMPORARY or ArchiveWriter::DONE_ERROR)
*
* @param Date $dateStart Only the month will be used
* @return int Returns the total number of rows deleted.
*/
public function purgeOutdatedArchives(Date $dateStart)
{
$purgeArchivesOlderThan = $this->getOldestTemporaryArchiveToKeepThreshold();
$deletedRowCount = 0;
$idArchivesToDelete = $this->getOutdatedArchiveIds($dateStart, $purgeArchivesOlderThan);
if (!empty($idArchivesToDelete)) {
$deletedRowCount = $this->deleteArchiveIds($dateStart, $idArchivesToDelete);
$this->logger->info("Deleted {count} rows in archive tables (numeric + blob) for {date}.", array(
'count' => $deletedRowCount,
'date' => $dateStart
));
} else {
$this->logger->debug("No outdated archives found in archive numeric table for {date}.", array('date' => $dateStart));
}
$this->logger->debug("Purging temporary archives: done [ purged archives older than {date} in {yearMonth} ] [Deleted IDs count: {deletedIds}]", array(
'date' => $purgeArchivesOlderThan,
'yearMonth' => $dateStart->toString('Y-m'),
'deletedIds' => count($idArchivesToDelete),
));
return $deletedRowCount;
}
public function purgeDeletedSiteArchives(Date $dateStart)
{
$archiveTable = ArchiveTableCreator::getNumericTable($dateStart);
$idArchivesToDelete = $this->model->getArchiveIdsForDeletedSites($archiveTable);
return $this->purge($idArchivesToDelete, $dateStart, 'deleted sites');
}
/**
* @param Date $dateStart
* @param array $deletedSegments List of segments whose archives should be purged
* @return int
*/
public function purgeDeletedSegmentArchives(Date $dateStart, array $deletedSegments)
{
if (count($deletedSegments)) {
$idArchivesToDelete = $this->getDeletedSegmentArchiveIds($dateStart, $deletedSegments);
return $this->purge($idArchivesToDelete, $dateStart, 'deleted segments');
}
}
/**
* Purge all numeric and blob archives with the given IDs from the database.
* @param array $idArchivesToDelete
* @param Date $dateStart
* @param string $reason
* @return int
*/
protected function purge(array $idArchivesToDelete, Date $dateStart, $reason)
{
$deletedRowCount = 0;
if (!empty($idArchivesToDelete)) {
$deletedRowCount = $this->deleteArchiveIds($dateStart, $idArchivesToDelete);
$this->logger->info(
"Deleted {count} rows in archive tables (numeric + blob) for {reason} for {date}.",
array(
'count' => $deletedRowCount,
'date' => $dateStart,
'reason' => $reason
)
);
$this->logger->debug("[Deleted IDs count: {deletedIds}]", array(
'deletedIds' => count($idArchivesToDelete),
));
} else {
$this->logger->debug(
"No archives for {reason} found in archive numeric table for {date}.",
array('date' => $dateStart, 'reason' => $reason)
);
}
return $deletedRowCount;
}
protected function getDeletedSegmentArchiveIds(Date $date, array $deletedSegments)
{
$archiveTable = ArchiveTableCreator::getNumericTable($date);
return $this->model->getArchiveIdsForSegments(
$archiveTable, $deletedSegments, $this->getOldestTemporaryArchiveToKeepThreshold()
);
}
protected function getOutdatedArchiveIds(Date $date, $purgeArchivesOlderThan)
{
$archiveTable = ArchiveTableCreator::getNumericTable($date);
$result = $this->model->getTemporaryArchivesOlderThan($archiveTable, $purgeArchivesOlderThan);
$idArchivesToDelete = array();
if (!empty($result)) {
foreach ($result as $row) {
$idArchivesToDelete[] = $row['idarchive'];
}
}
return $idArchivesToDelete;
}
/**
* Deleting "Custom Date Range" reports after 1 day, since they can be re-processed and would take up un-necessary space.
*
* @param $date Date
* @return int The total number of rows deleted from both the numeric & blob table.
*/
public function purgeArchivesWithPeriodRange(Date $date)
{
$numericTable = ArchiveTableCreator::getNumericTable($date);
$blobTable = ArchiveTableCreator::getBlobTable($date);
$deletedCount = $this->model->deleteArchivesWithPeriod(
$numericTable, $blobTable, Piwik::$idPeriods['range'], $this->purgeCustomRangesOlderThan);
$level = $deletedCount == 0 ? LogLevel::DEBUG : LogLevel::INFO;
$this->logger->log($level, "Purged {count} range archive rows from {numericTable} & {blobTable}.", array(
'count' => $deletedCount,
'numericTable' => $numericTable,
'blobTable' => $blobTable
));
$this->logger->debug(" [ purged archives older than {threshold} ]", array('threshold' => $this->purgeCustomRangesOlderThan));
return $deletedCount;
}
/**
* Deletes by batches Archive IDs in the specified month,
*
* @param Date $date
* @param $idArchivesToDelete
* @return int Number of rows deleted from both numeric + blob table.
*/
protected function deleteArchiveIds(Date $date, $idArchivesToDelete)
{
$batches = array_chunk($idArchivesToDelete, 1000);
$numericTable = ArchiveTableCreator::getNumericTable($date);
$blobTable = ArchiveTableCreator::getBlobTable($date);
$deletedCount = 0;
foreach ($batches as $idsToDelete) {
$deletedCount += $this->model->deleteArchiveIds($numericTable, $blobTable, $idsToDelete);
}
return $deletedCount;
}
/**
* Returns a timestamp indicating outdated archives older than this timestamp (processed before) can be purged.
*
* @return int|bool Outdated archives older than this timestamp should be purged
*/
protected function getOldestTemporaryArchiveToKeepThreshold()
{
$temporaryArchivingTimeout = Rules::getTodayArchiveTimeToLive();
if (Rules::isBrowserTriggerEnabled()) {
// If Browser Archiving is enabled, it is likely there are many more temporary archives
// We delete more often which is safe, since reports are re-processed on demand
return Date::factory($this->now - 2 * $temporaryArchivingTimeout)->getDateTime();
}
// If cron core:archive command is building the reports, we should keep all temporary reports from today
return $this->yesterday->getDateTime();
}
private static function getDefaultCustomRangeToPurgeAgeThreshold()
{
$daysRangesValid = Config::getInstance()->General['purge_date_range_archives_after_X_days'];
return Date::factory('today')->subDay($daysRangesValid)->getDateTime();
}
/**
* For tests.
*
* @param Date $yesterday
*/
public function setYesterdayDate(Date $yesterday)
{
$this->yesterday = $yesterday;
}
/**
* For tests.
*
* @param Date $today
*/
public function setTodayDate(Date $today)
{
$this->today = $today;
}
/**
* For tests.
*
* @param int $now
*/
public function setNow($now)
{
$this->now = $now;
}
}

49
matomo/core/Archive/ArchiveQuery.php Normal file
View file

@ -0,0 +1,49 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archive;
use Piwik\DataTable;
interface ArchiveQuery
{
/**
* @param string|string[] $names
* @return false|number|array
*/
public function getNumeric($names);
/**
* @param string|string[] $names
* @return DataTable|DataTable\Map
*/
public function getDataTableFromNumeric($names);
/**
* @param $names
* @return mixed
*/
public function getDataTableFromNumericAndMergeChildren($names);
/**
* @param string $name
* @param int|string|null $idSubtable
* @return DataTable|DataTable\Map
*/
public function getDataTable($name, $idSubtable = null);
/**
* @param string $name
* @param int|string|null $idSubtable
* @param int|null $depth
* @param bool $addMetadataSubtableId
* @return DataTable|DataTable\Map
*/
public function getDataTableExpanded($name, $idSubtable = null, $depth = null, $addMetadataSubtableId = true);
}

130
matomo/core/Archive/ArchiveQueryFactory.php Normal file
View file

@ -0,0 +1,130 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archive;
use Piwik\Archive;
use Piwik\Period;
use Piwik\Segment;
use Piwik\Site;
use Piwik\Period\Factory as PeriodFactory;
class ArchiveQueryFactory
{
public function __construct()
{
// empty
}
/**
* @see \Piwik\Archive::build()
*/
public function build($idSites, $strPeriod, $strDate, $strSegment = false, $_restrictSitesToLogin = false)
{
list($websiteIds, $timezone, $idSiteIsAll) = $this->getSiteInfoFromQueryParam($idSites, $_restrictSitesToLogin);
list($allPeriods, $isMultipleDate) = $this->getPeriodInfoFromQueryParam($strDate, $strPeriod, $timezone);
$segment = $this->getSegmentFromQueryParam($strSegment, $websiteIds, $allPeriods);
return $this->factory($segment, $allPeriods, $websiteIds, $idSiteIsAll, $isMultipleDate);
}
/**
* @see \Piwik\Archive::factory()
*/
public function factory(Segment $segment, array $periods, array $idSites, $idSiteIsAll = false, $isMultipleDate = false)
{
$forceIndexedBySite = false;
$forceIndexedByDate = false;
if ($idSiteIsAll || count($idSites) > 1) {
$forceIndexedBySite = true;
}
if (count($periods) > 1 || $isMultipleDate) {
$forceIndexedByDate = true;
}
$params = new Parameters($idSites, $periods, $segment);
return $this->newInstance($params, $forceIndexedBySite, $forceIndexedByDate);
}
public function newInstance(Parameters $params, $forceIndexedBySite, $forceIndexedByDate)
{
return new Archive($params, $forceIndexedBySite, $forceIndexedByDate);
}
/**
* Parses the site ID string provided in the 'idSite' query parameter to a list of
* website IDs.
*
* @param string $idSites the value of the 'idSite' query parameter
* @param bool $_restrictSitesToLogin
* @return array an array containing three elements:
* - an array of website IDs
* - string timezone to use (or false to use no timezone) when creating periods.
* - true if the request was for all websites (this forces the archive result to
* be indexed by site, even if there is only one site in Piwik)
*/
protected function getSiteInfoFromQueryParam($idSites, $_restrictSitesToLogin)
{
$websiteIds = Site::getIdSitesFromIdSitesString($idSites, $_restrictSitesToLogin);
$timezone = false;
if (count($websiteIds) === 1) {
$timezone = Site::getTimezoneFor($websiteIds[0]);
}
$idSiteIsAll = $idSites === Archive::REQUEST_ALL_WEBSITES_FLAG;
return [$websiteIds, $timezone, $idSiteIsAll];
}
/**
* Parses the date & period query parameters into a list of periods.
*
* @param string $strDate the value of the 'date' query parameter
* @param string $strPeriod the value of the 'period' query parameter
* @param string $timezone the timezone to use when constructing periods.
* @return array an array containing two elements:
* - the list of period objects to query archive data for
* - true if the request was for multiple periods (ie, two months, two weeks, etc.), false if otherwise.
* (this forces the archive result to be indexed by period, even if the list of periods
* has only one period).
*/
protected function getPeriodInfoFromQueryParam($strDate, $strPeriod, $timezone)
{
if (Period::isMultiplePeriod($strDate, $strPeriod)) {
$oPeriod = PeriodFactory::build($strPeriod, $strDate, $timezone);
$allPeriods = $oPeriod->getSubperiods();
} else {
$oPeriod = PeriodFactory::makePeriodFromQueryParams($timezone, $strPeriod, $strDate);
$allPeriods = array($oPeriod);
}
$isMultipleDate = Period::isMultiplePeriod($strDate, $strPeriod);
return [$allPeriods, $isMultipleDate];
}
/**
* Parses the segment query parameter into a Segment object.
*
* @param string $strSegment the value of the 'segment' query parameter.
* @param int[] $websiteIds the list of sites being queried.
* @param Period[] $allPeriods list of all periods
* @return Segment
*/
protected function getSegmentFromQueryParam($strSegment, $websiteIds, $allPeriods)
{
// we might have multiple periods, so use the start date of the first one and
// the end date of the last one to limit the possible segment subquery
return new Segment($strSegment, $websiteIds, reset($allPeriods)->getDateTimeStart(), end($allPeriods)->getDateTimeEnd());
}
}

142
matomo/core/Archive/Chunk.php Normal file
View file

@ -0,0 +1,142 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archive;
/**
* This class is used to split blobs of DataTables into chunks. Each blob used to be stored under one blob in the
* archive table. For better efficiency we do now combine multiple DataTable into one blob entry.
*
* Chunks are identified by having the recordName $recordName_chunk_0_99, $recordName_chunk_100_199 (this chunk stores
* the subtable 100-199).
*/
class Chunk
{
const ARCHIVE_APPENDIX_SUBTABLES = 'chunk';
const NUM_TABLES_IN_CHUNK = 100;
/**
* Gets the record name to use for a given tableId/subtableId.
*
* @param string $recordName eg 'Actions_ActionsUrl'
* @param int $tableId eg '5' for tableId '5'
* @return string eg 'Actions_ActionsUrl_chunk_0_99' as the table should be stored under this blob id.
*/
public function getRecordNameForTableId($recordName, $tableId)
{
$chunk = (floor($tableId / self::NUM_TABLES_IN_CHUNK));
$start = $chunk * self::NUM_TABLES_IN_CHUNK;
$end = $start + self::NUM_TABLES_IN_CHUNK - 1;
return $recordName . $this->getAppendix() . $start . '_' . $end;
}
/**
* Moves the given blobs into chunks and assigns a proper record name containing the chunk number.
*
* @param string $recordName The original archive record name, eg 'Actions_ActionsUrl'
* @param array $blobs An array containing a mapping of tableIds to blobs. Eg array(0 => 'blob', 1 => 'subtableBlob', ...)
* @return array An array where each blob is moved into a chunk, indexed by recordNames.
* eg array('Actions_ActionsUrl_chunk_0_99' => array(0 => 'blob', 1 => 'subtableBlob', ...),
* 'Actions_ActionsUrl_chunk_100_199' => array(...))
*/
public function moveArchiveBlobsIntoChunks($recordName, $blobs)
{
$chunks = array();
foreach ($blobs as $tableId => $blob) {
$name = $this->getRecordNameForTableId($recordName, $tableId);
if (!array_key_exists($name, $chunks)) {
$chunks[$name] = array();
}
$chunks[$name][$tableId] = $blob;
}
return $chunks;
}
/**
* Detects whether a recordName like 'Actions_ActionUrls_chunk_0_99' or 'Actions_ActionUrls' belongs to a
* chunk or not.
*
* To be a valid recordName that belongs to a chunk it must end with '_chunk_NUMERIC_NUMERIC'.
*
* @param string $recordName
* @return bool
*/
public function isRecordNameAChunk($recordName)
{
$posAppendix = $this->getEndPosOfChunkAppendix($recordName);
if (false === $posAppendix) {
return false;
}
// will contain "0_99" of "chunk_0_99"
$blobId = substr($recordName, $posAppendix);
return $this->isChunkRange($blobId);
}
private function isChunkRange($blobId)
{
$blobId = explode('_', $blobId);
return 2 === count($blobId) && is_numeric($blobId[0]) && is_numeric($blobId[1]);
}
/**
* When having a record like 'Actions_ActionUrls_chunk_0_99" it will return the raw recordName 'Actions_ActionUrls'.
*
* @param string $recordName
* @return string
*/
public function getRecordNameWithoutChunkAppendix($recordName)
{
if (!$this->isRecordNameAChunk($recordName)) {
return $recordName;
}
$posAppendix = $this->getStartPosOfChunkAppendix($recordName);
if (false === $posAppendix) {
return $recordName;
}
return substr($recordName, 0, $posAppendix);
}
/**
* Returns the string that is appended to the original record name. This appendix identifes a record name is a
* chunk.
* @return string
*/
public function getAppendix()
{
return '_' . self::ARCHIVE_APPENDIX_SUBTABLES . '_';
}
private function getStartPosOfChunkAppendix($recordName)
{
return strpos($recordName, $this->getAppendix());
}
private function getEndPosOfChunkAppendix($recordName)
{
$pos = strpos($recordName, $this->getAppendix());
if ($pos === false) {
return false;
}
return $pos + strlen($this->getAppendix());
}
}

417
matomo/core/Archive/DataCollection.php Normal file
View file

@ -0,0 +1,417 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archive;
use Exception;
use Piwik\DataTable;
/**
* This class is used to hold and transform archive data for the Archive class.
*
* Archive data is loaded into an instance of this type, can be indexed by archive
* metadata (such as the site ID, period string, etc.), and can be transformed into
* DataTable and Map instances.
*/
class DataCollection
{
const METADATA_CONTAINER_ROW_KEY = '_metadata';
/**
* The archive data, indexed first by site ID and then by period date range. Eg,
*
* array(
* '0' => array(
* array(
* '2012-01-01,2012-01-01' => array(...),
* '2012-01-02,2012-01-02' => array(...),
* )
* ),
* '1' => array(
* array(
* '2012-01-01,2012-01-01' => array(...),
* )
* )
* )
*
* Archive data can be either a numeric value or a serialized string blob. Every
* piece of archive data is associated by it's archive name. For example,
* the array(...) above could look like:
*
* array(
* 'nb_visits' => 1,
* 'nb_actions' => 2
* )
*
* There is a special element '_metadata' in data rows that holds values treated
* as DataTable metadata.
*/
private $data = array();
/**
* The whole list of metric/record names that were used in the archive query.
*
* @var array
*/
private $dataNames;
/**
* The type of data that was queried for (ie, "blob" or "numeric").
*
* @var string
*/
private $dataType;
/**
* The default values to use for each metric/record name that's being queried
* for.
*
* @var array
*/
private $defaultRow;
/**
* The list of all site IDs that were queried for.
*
* @var array
*/
private $sitesId;
/**
* The list of all periods that were queried for. Each period is associated with
* the period's range string. Eg,
*
* array(
* '2012-01-01,2012-01-31' => new Period(...),
* '2012-02-01,2012-02-28' => new Period(...),
* )
*
* @var \Piwik\Period[]
*/
private $periods;
/**
* Constructor.
*
* @param array $dataNames @see $this->dataNames
* @param string $dataType @see $this->dataType
* @param array $sitesId @see $this->sitesId
* @param \Piwik\Period[] $periods @see $this->periods
* @param array $defaultRow @see $this->defaultRow
*/
public function __construct($dataNames, $dataType, $sitesId, $periods, $segment, $defaultRow = null)
{
$this->dataNames = $dataNames;
$this->dataType = $dataType;
if ($defaultRow === null) {
$defaultRow = array_fill_keys($dataNames, 0);
}
$this->sitesId = $sitesId;
foreach ($periods as $period) {
$this->periods[$period->getRangeString()] = $period;
}
$this->segment = $segment;
$this->defaultRow = $defaultRow;
}
/**
* Returns a reference to the data for a specific site & period. If there is
* no data for the given site ID & period, it is set to the default row.
*
* @param int $idSite
* @param string $period eg, '2012-01-01,2012-01-31'
*/
public function &get($idSite, $period)
{
if (!isset($this->data[$idSite][$period])) {
$this->data[$idSite][$period] = $this->defaultRow;
}
return $this->data[$idSite][$period];
}
/**
* Set data for a specific site & period. If there is no data for the given site ID & period,
* it is set to the default row.
*
* @param int $idSite
* @param string $period eg, '2012-01-01,2012-01-31'
* @param string $name eg 'nb_visits'
* @param string $value eg 5
* @param array $meta Optional metadata to add to the row
*/
public function set($idSite, $period, $name, $value, array $meta = null)
{
$row = & $this->get($idSite, $period);
$row[$name] = $value;
if ($meta) {
foreach ($meta as $k => $v) {
$row[self::METADATA_CONTAINER_ROW_KEY][$k] = $v;
}
}
}
/**
* Adds a new metadata to the data for specific site & period. If there is no
* data for the given site ID & period, it is set to the default row.
*
* Note: Site ID and period range string are two special types of metadata. Since
* the data stored in this class is indexed by site & period, this metadata is not
* stored in individual data rows.
*
* @param int $idSite
* @param string $period eg, '2012-01-01,2012-01-31'
* @param string $name The metadata name.
* @param mixed $value The metadata name.
*/
public function addMetadata($idSite, $period, $name, $value)
{
$row = & $this->get($idSite, $period);
$row[self::METADATA_CONTAINER_ROW_KEY][$name] = $value;
}
/**
* Returns archive data as an array indexed by metadata.
*
* @param array $resultIndices An array mapping metadata names to pretty labels
* for them. Each archive data row will be indexed
* by the metadata specified here.
*
* Eg, array('site' => 'idSite', 'period' => 'Date')
* @return array
*/
public function getIndexedArray($resultIndices)
{
$indexKeys = array_keys($resultIndices);
$result = $this->createOrderedIndex($indexKeys);
foreach ($this->data as $idSite => $rowsByPeriod) {
foreach ($rowsByPeriod as $period => $row) {
// FIXME: This hack works around a strange bug that occurs when getting
// archive IDs through ArchiveProcessing instances. When a table
// does not already exist, for some reason the archive ID for
// today (or from two days ago) will be added to the Archive
// instances list. The Archive instance will then select data
// for periods outside of the requested set.
// working around the bug here, but ideally, we need to figure
// out why incorrect idarchives are being selected.
if (empty($this->periods[$period])) {
continue;
}
$this->putRowInIndex($result, $indexKeys, $row, $idSite, $period);
}
}
return $result;
}
/**
* Returns archive data as a DataTable indexed by metadata. Indexed data will
* be represented by Map instances.
*
* @param array $resultIndices An array mapping metadata names to pretty labels
* for them. Each archive data row will be indexed
* by the metadata specified here.
*
* Eg, array('site' => 'idSite', 'period' => 'Date')
* @return DataTable|DataTable\Map
*/
public function getDataTable($resultIndices)
{
$dataTableFactory = new DataTableFactory(
$this->dataNames, $this->dataType, $this->sitesId, $this->periods, $this->segment, $this->defaultRow);
$index = $this->getIndexedArray($resultIndices);
return $dataTableFactory->make($index, $resultIndices);
}
/**
* See {@link DataTableFactory::makeMerged()}
*
* @param array $resultIndices
* @return DataTable|DataTable\Map
* @throws Exception
*/
public function getMergedDataTable($resultIndices)
{
$dataTableFactory = new DataTableFactory(
$this->dataNames, $this->dataType, $this->sitesId, $this->periods, $this->segment, $this->defaultRow);
$index = $this->getIndexedArray($resultIndices);
return $dataTableFactory->makeMerged($index, $resultIndices);
}
/**
* Returns archive data as a DataTable indexed by metadata. Indexed data will
* be represented by Map instances. Each DataTable will have
* its subtable IDs set.
*
* This function will only work if blob data was loaded and only one record
* was loaded (not including subtables of the record).
*
* @param array $resultIndices An array mapping metadata names to pretty labels
* for them. Each archive data row will be indexed
* by the metadata specified here.
*
* Eg, array('site' => 'idSite', 'period' => 'Date')
* @param int|null $idSubTable The subtable to return.
* @param int|null $depth max depth for subtables.
* @param bool $addMetadataSubTableId Whether to add the DB subtable ID as metadata
* to each datatable, or not.
* @throws Exception
* @return DataTable|DataTable\Map
*/
public function getExpandedDataTable($resultIndices, $idSubTable = null, $depth = null, $addMetadataSubTableId = false)
{
$this->checkExpandedMethodPrerequisites();
$dataTableFactory = new DataTableFactory(
$this->dataNames, 'blob', $this->sitesId, $this->periods, $this->segment, $this->defaultRow);
$dataTableFactory->expandDataTable($depth, $addMetadataSubTableId);
$dataTableFactory->useSubtable($idSubTable);
$index = $this->getIndexedArray($resultIndices);
return $dataTableFactory->make($index, $resultIndices);
}
/**
* Returns metadata for a data row.
*
* @param array $data The data row.
* @return array
*/
public static function getDataRowMetadata($data)
{
if (isset($data[self::METADATA_CONTAINER_ROW_KEY])) {
return $data[self::METADATA_CONTAINER_ROW_KEY];
} else {
return array();
}
}
/**
* Removes all table metadata from a data row.
*
* @param array $data The data row.
*/
public static function removeMetadataFromDataRow(&$data)
{
unset($data[self::METADATA_CONTAINER_ROW_KEY]);
}
/**
* Creates an empty index using a list of metadata names. If the 'site' and/or
* 'period' metadata names are supplied, empty rows are added for every site/period
* that was queried for.
*
* Using this function ensures consistent ordering in the indexed result.
*
* @param array $metadataNamesToIndexBy List of metadata names to index archive data by.
* @return array
*/
private function createOrderedIndex($metadataNamesToIndexBy)
{
$result = array();
if (!empty($metadataNamesToIndexBy)) {
$metadataName = array_shift($metadataNamesToIndexBy);
if ($metadataName == DataTableFactory::TABLE_METADATA_SITE_INDEX) {
$indexKeyValues = array_values($this->sitesId);
} elseif ($metadataName == DataTableFactory::TABLE_METADATA_PERIOD_INDEX) {
$indexKeyValues = array_keys($this->periods);
}
if (empty($metadataNamesToIndexBy)) {
$result = array_fill_keys($indexKeyValues, array());
} else {
foreach ($indexKeyValues as $key) {
$result[$key] = $this->createOrderedIndex($metadataNamesToIndexBy);
}
}
}
return $result;
}
/**
* Puts an archive data row in an index.
*/
private function putRowInIndex(&$index, $metadataNamesToIndexBy, $row, $idSite, $period)
{
$currentLevel = & $index;
foreach ($metadataNamesToIndexBy as $metadataName) {
if ($metadataName == DataTableFactory::TABLE_METADATA_SITE_INDEX) {
$key = $idSite;
} elseif ($metadataName == DataTableFactory::TABLE_METADATA_PERIOD_INDEX) {
$key = $period;
} else {
$key = $row[self::METADATA_CONTAINER_ROW_KEY][$metadataName];
}
if (!isset($currentLevel[$key])) {
$currentLevel[$key] = array();
}
$currentLevel = & $currentLevel[$key];
}
$currentLevel = $row;
}
public function forEachBlobExpanded($callable, $idSubTable = null, $depth = null, $addMetadataSubTableId = false)
{
$this->checkExpandedMethodPrerequisites();
$dataTableFactory = new DataTableFactory(
$this->dataNames, 'blob', $this->sitesId, $this->periods, $this->segment, $this->defaultRow);
$dataTableFactory->expandDataTable($depth, $addMetadataSubTableId);
$dataTableFactory->useSubtable($idSubTable);
foreach ($this->data as $idSite => $periods) {
foreach ($periods as $periodRange => $data) {
// FIXME: This hack works around a strange bug that occurs when getting
// archive IDs through ArchiveProcessing instances. When a table
// does not already exist, for some reason the archive ID for
// today (or from two days ago) will be added to the Archive
// instances list. The Archive instance will then select data
// for periods outside of the requested set.
// working around the bug here, but ideally, we need to figure
// out why incorrect idarchives are being selected.
if (empty($this->periods[$periodRange])) {
continue;
}
$tableMetadata = $dataTableFactory->getTableMetadataFor($idSite, $this->periods[$periodRange]);
$callable($data, $dataTableFactory, $tableMetadata);
}
}
}
private function checkExpandedMethodPrerequisites()
{
if ($this->dataType != 'blob') {
throw new Exception("DataCollection: cannot call getExpandedDataTable with "
. "{$this->dataType} data types. Only works with blob data.");
}
if (count($this->dataNames) !== 1) {
throw new Exception("DataCollection: cannot call getExpandedDataTable with "
. "more than one record.");
}
}
}

610
matomo/core/Archive/DataTableFactory.php Normal file
View file

@ -0,0 +1,610 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archive;
use Piwik\DataTable;
use Piwik\DataTable\Row;
use Piwik\Segment;
use Piwik\Site;
/**
* Creates a DataTable or Set instance based on an array
* index created by DataCollection.
*
* This class is only used by DataCollection.
*/
class DataTableFactory
{
const TABLE_METADATA_SEGMENT_INDEX = 'segment';
const TABLE_METADATA_SEGMENT_PRETTY_INDEX = 'segmentPretty';
/**
* @see DataCollection::$dataNames.
*/
private $dataNames;
/**
* @see DataCollection::$dataType.
*/
private $dataType;
/**
* Whether to expand the DataTables that're created or not. Expanding a DataTable
* means creating DataTables using subtable blobs and correctly setting the subtable
* IDs of all DataTables.
*
* @var bool
*/
private $expandDataTable = false;
/**
* Whether to add the subtable ID used in the database to the in-memory DataTables
* as metadata or not.
*
* @var bool
*/
private $addMetadataSubtableId = false;
/**
* The maximum number of subtable levels to create when creating an expanded
* DataTable.
*
* @var int
*/
private $maxSubtableDepth = null;
/**
* @see DataCollection::$sitesId.
*/
private $sitesId;
/**
* @see DataCollection::$periods.
*/
private $periods;
/**
* @var Segment
*/
private $segment;
/**
* The ID of the subtable to create a DataTable for. Only relevant for blob data.
*
* @var int|null
*/
private $idSubtable = null;
/**
* @see DataCollection::$defaultRow.
*/
private $defaultRow;
const TABLE_METADATA_SITE_INDEX = 'site';
const TABLE_METADATA_PERIOD_INDEX = 'period';
/**
* Constructor.
*/
public function __construct($dataNames, $dataType, $sitesId, $periods, Segment $segment, $defaultRow)
{
$this->dataNames = $dataNames;
$this->dataType = $dataType;
$this->sitesId = $sitesId;
//here index period by string only
$this->periods = $periods;
$this->segment = $segment;
$this->defaultRow = $defaultRow;
}
/**
* Returns the ID of the site a table is related to based on the 'site' metadata entry,
* or null if there is none.
*
* @param DataTable $table
* @return int|null
*/
public static function getSiteIdFromMetadata(DataTable $table)
{
$site = $table->getMetadata(self::TABLE_METADATA_SITE_INDEX);
if (empty($site)) {
return null;
} else {
return $site->getId();
}
}
/**
* Tells the factory instance to expand the DataTables that are created by
* creating subtables and setting the subtable IDs of rows w/ subtables correctly.
*
* @param null|int $maxSubtableDepth max depth for subtables.
* @param bool $addMetadataSubtableId Whether to add the subtable ID used in the
* database to the in-memory DataTables as
* metadata or not.
*/
public function expandDataTable($maxSubtableDepth = null, $addMetadataSubtableId = false)
{
$this->expandDataTable = true;
$this->maxSubtableDepth = $maxSubtableDepth;
$this->addMetadataSubtableId = $addMetadataSubtableId;
}
/**
* Tells the factory instance to create a DataTable using a blob with the
* supplied subtable ID.
*
* @param int $idSubtable An in-database subtable ID.
* @throws \Exception
*/
public function useSubtable($idSubtable)
{
if (count($this->dataNames) !== 1) {
throw new \Exception("DataTableFactory: Getting subtables for multiple records in one"
. " archive query is not currently supported.");
}
$this->idSubtable = $idSubtable;
}
private function isNumericDataType()
{
return $this->dataType == 'numeric';
}
/**
* Creates a DataTable|Set instance using an index of
* archive data.
*
* @param array $index @see DataCollection
* @param array $resultIndices an array mapping metadata names with pretty metadata
* labels.
* @return DataTable|DataTable\Map
*/
public function make($index, $resultIndices, $keyMetadata = null)
{
$keyMetadata = $keyMetadata ?: $this->getDefaultMetadata();
if (empty($resultIndices)) {
// for numeric data, if there's no index (and thus only 1 site & period in the query),
// we want to display every queried metric name
if (empty($index)
&& $this->isNumericDataType()
) {
$index = $this->defaultRow;
}
$dataTable = $this->createDataTable($index, $keyMetadata);
} else {
$dataTable = $this->createDataTableMapFromIndex($index, $resultIndices, $keyMetadata);
}
return $dataTable;
}
/**
* Creates a merged DataTable|Map instance using an index of archive data similar to {@link make()}.
*
* Whereas {@link make()} creates a Map for each result index (period and|or site), this will only create a Map
* for a period result index and move all site related indices into one dataTable. This is the same as doing
* `$dataTableFactory->make()->mergeChildren()` just much faster. It is mainly useful for reports across many sites
* eg `MultiSites.getAll`. Was done as part of https://github.com/piwik/piwik/issues/6809
*
* @param array $index @see DataCollection
* @param array $resultIndices an array mapping metadata names with pretty metadata labels.
*
* @return DataTable|DataTable\Map
* @throws \Exception
*/
public function makeMerged($index, $resultIndices)
{
if (!$this->isNumericDataType()) {
throw new \Exception('This method is supposed to work with non-numeric data types but it is not tested. To use it, remove this exception and write tests to be sure it works.');
}
$hasSiteIndex = isset($resultIndices[self::TABLE_METADATA_SITE_INDEX]);
$hasPeriodIndex = isset($resultIndices[self::TABLE_METADATA_PERIOD_INDEX]);
$isNumeric = $this->isNumericDataType();
// to be backwards compatible use a Simple table if needed as it will be formatted differently
$useSimpleDataTable = !$hasSiteIndex && $isNumeric;
if (!$hasSiteIndex) {
$firstIdSite = reset($this->sitesId);
$index = array($firstIdSite => $index);
}
if ($hasPeriodIndex) {
$dataTable = $this->makeMergedTableWithPeriodAndSiteIndex($index, $resultIndices, $useSimpleDataTable, $isNumeric);
} else {
$dataTable = $this->makeMergedWithSiteIndex($index, $useSimpleDataTable, $isNumeric);
}
return $dataTable;
}
/**
* Creates a DataTable|Set instance using an array
* of blobs.
*
* If only one record is being queried, a single DataTable will
* be returned. Otherwise, a DataTable\Map is returned that indexes
* DataTables by record name.
*
* If expandDataTable was called, and only one record is being queried,
* the created DataTable's subtables will be expanded.
*
* @param array $blobRow
* @return DataTable|DataTable\Map
*/
private function makeFromBlobRow($blobRow, $keyMetadata)
{
if ($blobRow === false) {
$table = new DataTable();
$table->setAllTableMetadata($keyMetadata);
$this->setPrettySegmentMetadata($table);
return $table;
}
if (count($this->dataNames) === 1) {
return $this->makeDataTableFromSingleBlob($blobRow, $keyMetadata);
} else {
return $this->makeIndexedByRecordNameDataTable($blobRow, $keyMetadata);
}
}
/**
* Creates a DataTable for one record from an archive data row.
*
* @see makeFromBlobRow
*
* @param array $blobRow
* @return DataTable
*/
private function makeDataTableFromSingleBlob($blobRow, $keyMetadata)
{
$recordName = reset($this->dataNames);
if ($this->idSubtable !== null) {
$recordName .= '_' . $this->idSubtable;
}
if (!empty($blobRow[$recordName])) {
$table = DataTable::fromSerializedArray($blobRow[$recordName]);
} else {
$table = new DataTable();
}
// set table metadata
$table->setAllTableMetadata(array_merge($table->getAllTableMetadata(), DataCollection::getDataRowMetadata($blobRow), $keyMetadata));
$this->setPrettySegmentMetadata($table);
if ($this->expandDataTable) {
$table->enableRecursiveFilters();
$this->setSubtables($table, $blobRow);
}
return $table;
}
/**
* Creates a DataTable for every record in an archive data row and puts them
* in a DataTable\Map instance.
*
* @param array $blobRow
* @return DataTable\Map
*/
private function makeIndexedByRecordNameDataTable($blobRow, $keyMetadata)
{
$table = new DataTable\Map();
$table->setKeyName('recordName');
$tableMetadata = array_merge(DataCollection::getDataRowMetadata($blobRow), $keyMetadata);
foreach ($blobRow as $name => $blob) {
$newTable = DataTable::fromSerializedArray($blob);
$newTable->setAllTableMetadata(array_merge($newTable->getAllTableMetadata(), $tableMetadata));
$this->setPrettySegmentMetadata($newTable);
$table->addTable($newTable, $name);
}
return $table;
}
/**
* Creates a Set from an array index.
*
* @param array $index @see DataCollection
* @param array $resultIndices @see make
* @param array $keyMetadata The metadata to add to the table when it's created.
* @return DataTable\Map
*/
private function createDataTableMapFromIndex($index, $resultIndices, $keyMetadata)
{
$result = new DataTable\Map();
$result->setKeyName(reset($resultIndices));
$resultIndex = key($resultIndices);
array_shift($resultIndices);
$hasIndices = !empty($resultIndices);
foreach ($index as $label => $value) {
$keyMetadata[$resultIndex] = $this->createTableIndexMetadata($resultIndex, $label);
if ($hasIndices) {
$newTable = $this->createDataTableMapFromIndex($value, $resultIndices, $keyMetadata);
} else {
$newTable = $this->createDataTable($value, $keyMetadata);
}
$result->addTable($newTable, $this->prettifyIndexLabel($resultIndex, $label));
}
return $result;
}
private function createTableIndexMetadata($resultIndex, $label)
{
if ($resultIndex === DataTableFactory::TABLE_METADATA_SITE_INDEX) {
return new Site($label);
} elseif ($resultIndex === DataTableFactory::TABLE_METADATA_PERIOD_INDEX) {
return $this->periods[$label];
}
}
/**
* Creates a DataTable instance from an index row.
*
* @param array $data An archive data row.
* @param array $keyMetadata The metadata to add to the table(s) when created.
* @return DataTable|DataTable\Map
*/
private function createDataTable($data, $keyMetadata)
{
if ($this->dataType == 'blob') {
$result = $this->makeFromBlobRow($data, $keyMetadata);
} else {
$result = $this->makeFromMetricsArray($data, $keyMetadata);
}
return $result;
}
/**
* Creates DataTables from $dataTable's subtable blobs (stored in $blobRow) and sets
* the subtable IDs of each DataTable row.
*
* @param DataTable $dataTable
* @param array $blobRow An array associating record names (w/ subtable if applicable)
* with blob values. This should hold every subtable blob for
* the loaded DataTable.
* @param int $treeLevel
*/
private function setSubtables($dataTable, $blobRow, $treeLevel = 0)
{
if ($this->maxSubtableDepth
&& $treeLevel >= $this->maxSubtableDepth
) {
// unset the subtables so DataTableManager doesn't throw
foreach ($dataTable->getRowsWithoutSummaryRow() as $row) {
$row->removeSubtable();
}
$summaryRow = $dataTable->getRowFromId(DataTable::ID_SUMMARY_ROW);
if ($summaryRow) {
$summaryRow->removeSubtable();
}
return;
}
$dataName = reset($this->dataNames);
foreach ($dataTable->getRows() as $row) {
$sid = $row->getIdSubDataTable();
if ($sid === null) {
continue;
}
$blobName = $dataName . "_" . $sid;
if (!empty($blobRow[$blobName])) {
$subtable = DataTable::fromSerializedArray($blobRow[$blobName]);
$subtable->setMetadata(self::TABLE_METADATA_PERIOD_INDEX, $dataTable->getMetadata(self::TABLE_METADATA_PERIOD_INDEX));
$subtable->setMetadata(self::TABLE_METADATA_SITE_INDEX, $dataTable->getMetadata(self::TABLE_METADATA_SITE_INDEX));
$subtable->setMetadata(self::TABLE_METADATA_SEGMENT_INDEX, $dataTable->getMetadata(self::TABLE_METADATA_SEGMENT_INDEX));
$subtable->setMetadata(self::TABLE_METADATA_SEGMENT_PRETTY_INDEX, $dataTable->getMetadata(self::TABLE_METADATA_SEGMENT_PRETTY_INDEX));
$subtable->setMetadata(DataTable::ARCHIVED_DATE_METADATA_NAME, $dataTable->getMetadata(DataTable::ARCHIVED_DATE_METADATA_NAME));
$this->setSubtables($subtable, $blobRow, $treeLevel + 1);
// we edit the subtable ID so that it matches the newly table created in memory
// NB: we don't overwrite the datatableid in the case we are displaying the table expanded.
if ($this->addMetadataSubtableId) {
// this will be written back to the column 'idsubdatatable' just before rendering,
// see Renderer/Php.php
$row->addMetadata('idsubdatatable_in_db', $row->getIdSubDataTable());
}
$row->setSubtable($subtable);
}
}
}
private function getDefaultMetadata()
{
return array(
DataTableFactory::TABLE_METADATA_SITE_INDEX => new Site(reset($this->sitesId)),
DataTableFactory::TABLE_METADATA_PERIOD_INDEX => reset($this->periods),
DataTableFactory::TABLE_METADATA_SEGMENT_INDEX => $this->segment->getString(),
DataTableFactory::TABLE_METADATA_SEGMENT_PRETTY_INDEX => $this->segment->getString(),
);
}
public function getTableMetadataFor($idSite, $period)
{
return [
DataTableFactory::TABLE_METADATA_SITE_INDEX => new Site($idSite),
DataTableFactory::TABLE_METADATA_PERIOD_INDEX => $period,
DataTableFactory::TABLE_METADATA_SEGMENT_INDEX => $this->segment->getString(),
DataTableFactory::TABLE_METADATA_SEGMENT_PRETTY_INDEX => $this->segment->getString(),
];
}
/**
* Returns the pretty version of an index label.
*
* @param string $labelType eg, 'site', 'period', etc.
* @param string $label eg, '0', '1', '2012-01-01,2012-01-31', etc.
* @return string
*/
private function prettifyIndexLabel($labelType, $label)
{
if ($labelType == self::TABLE_METADATA_PERIOD_INDEX) { // prettify period labels
$period = $this->periods[$label];
$label = $period->getLabel();
if ($label === 'week' || $label === 'range') {
return $period->getRangeString();
}
return $period->getPrettyString();
}
return $label;
}
/**
* @param $data
* @return DataTable\Simple
*/
private function makeFromMetricsArray($data, $keyMetadata)
{
$table = new DataTable\Simple();
if (!empty($data)) {
$table->setAllTableMetadata(array_merge($table->getAllTableMetadata(), DataCollection::getDataRowMetadata($data), $keyMetadata));
$this->setPrettySegmentMetadata($table);
DataCollection::removeMetadataFromDataRow($data);
$table->addRow(new Row(array(Row::COLUMNS => $data)));
} else {
// if we're querying numeric data, we couldn't find any, and we're only
// looking for one metric, add a row w/ one column w/ value 0. this is to
// ensure that the PHP renderer outputs 0 when only one column is queried.
// w/o this code, an empty array would be created, and other parts of Piwik
// would break.
if (count($this->dataNames) == 1
&& $this->isNumericDataType()
) {
$name = reset($this->dataNames);
$table->addRow(new Row(array(Row::COLUMNS => array($name => 0))));
}
$table->setAllTableMetadata(array_merge($table->getAllTableMetadata(), $keyMetadata));
$this->setPrettySegmentMetadata($table);
}
$result = $table;
return $result;
}
private function makeMergedTableWithPeriodAndSiteIndex($index, $resultIndices, $useSimpleDataTable, $isNumeric)
{
$map = new DataTable\Map();
$map->setKeyName($resultIndices[self::TABLE_METADATA_PERIOD_INDEX]);
// we save all tables of the map in this array to be able to add rows fast
$tables = array();
foreach ($this->periods as $range => $period) {
// as the resulting table is "merged", we do only set Period metedata and no metadata for site. Instead each
// row will have an idsite metadata entry.
$metadata = array(self::TABLE_METADATA_PERIOD_INDEX => $period);
if ($useSimpleDataTable) {
$table = new DataTable\Simple();
} else {
$table = new DataTable();
}
$table->setAllTableMetadata(array_merge($table->getAllTableMetadata(), $metadata));
$this->setPrettySegmentMetadata($table);
$map->addTable($table, $this->prettifyIndexLabel(self::TABLE_METADATA_PERIOD_INDEX, $range));
$tables[$range] = $table;
}
foreach ($index as $idsite => $table) {
$rowMeta = array('idsite' => $idsite);
foreach ($table as $range => $row) {
if (!empty($row)) {
$tables[$range]->addRow(new Row(array(
Row::COLUMNS => $row,
Row::METADATA => $rowMeta)
));
} elseif ($isNumeric) {
$tables[$range]->addRow(new Row(array(
Row::COLUMNS => $this->defaultRow,
Row::METADATA => $rowMeta)
));
}
}
}
return $map;
}
private function makeMergedWithSiteIndex($index, $useSimpleDataTable, $isNumeric)
{
if ($useSimpleDataTable) {
$table = new DataTable\Simple();
} else {
$table = new DataTable();
}
$table->setAllTableMetadata(array(DataTableFactory::TABLE_METADATA_PERIOD_INDEX => reset($this->periods)));
$this->setPrettySegmentMetadata($table);
foreach ($index as $idsite => $row) {
$meta = array();
if (isset($row[DataCollection::METADATA_CONTAINER_ROW_KEY])) {
$meta = $row[DataCollection::METADATA_CONTAINER_ROW_KEY];
}
$meta['idsite'] = $idsite;
if (!empty($row)) {
$table->addRow(new Row(array(
Row::COLUMNS => $row,
Row::METADATA => $meta)
));
} elseif ($isNumeric) {
$table->addRow(new Row(array(
Row::COLUMNS => $this->defaultRow,
Row::METADATA => $meta)
));
}
}
return $table;
}
private function setPrettySegmentMetadata(DataTable $table)
{
$site = $table->getMetadata(self::TABLE_METADATA_SITE_INDEX);
$idSite = $site ? $site->getId() : false;
$segmentPretty = $this->segment->getStoredSegmentName($idSite);
$table->setMetadata('segment', $this->segment->getString());
$table->setMetadata('segmentPretty', $segmentPretty);
}
}

59
matomo/core/Archive/Parameters.php Normal file
View file

@ -0,0 +1,59 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archive;
use Piwik\Period;
use Piwik\Segment;
class Parameters
{
/**
* The list of site IDs to query archive data for.
*
* @var array
*/
private $idSites = array();
/**
* The list of Period's to query archive data for.
*
* @var Period[]
*/
private $periods = array();
/**
* Segment applied to the visits set.
*
* @var Segment
*/
private $segment;
public function getSegment()
{
return $this->segment;
}
public function __construct($idSites, $periods, Segment $segment)
{
$this->idSites = $idSites;
$this->periods = $periods;
$this->segment = $segment;
}
public function getPeriods()
{
return $this->periods;
}
public function getIdSites()
{
return $this->idSites;
}
}

706
matomo/core/ArchiveProcessor.php Normal file
View file

@ -0,0 +1,706 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
use Exception;
use Piwik\Archive\DataCollection;
use Piwik\Archive\DataTableFactory;
use Piwik\ArchiveProcessor\Parameters;
use Piwik\ArchiveProcessor\Rules;
use Piwik\Container\StaticContainer;
use Piwik\DataAccess\ArchiveWriter;
use Piwik\DataAccess\LogAggregator;
use Piwik\DataTable\Manager;
use Piwik\DataTable\Map;
use Piwik\DataTable\Row;
use Piwik\Segment\SegmentExpression;
use Psr\Log\LoggerInterface;
/**
* Used by {@link Piwik\Plugin\Archiver} instances to insert and aggregate archive data.
*
* ### See also
*
* - **{@link Piwik\Plugin\Archiver}** - to learn how plugins should implement their own analytics
* aggregation logic.
* - **{@link Piwik\DataAccess\LogAggregator}** - to learn how plugins can perform data aggregation
* across Piwik's log tables.
*
* ### Examples
*
* **Inserting numeric data**
*
* // function in an Archiver descendant
* public function aggregateDayReport()
* {
* $archiveProcessor = $this->getProcessor();
*
* $myFancyMetric = // ... calculate the metric value ...
* $archiveProcessor->insertNumericRecord('MyPlugin_myFancyMetric', $myFancyMetric);
* }
*
* **Inserting serialized DataTables**
*
* // function in an Archiver descendant
* public function aggregateDayReport()
* {
* $archiveProcessor = $this->getProcessor();
*
* $maxRowsInTable = Config::getInstance()->General['datatable_archiving_maximum_rows_standard'];j
*
* $dataTable = // ... build by aggregating visits ...
* $serializedData = $dataTable->getSerialized($maxRowsInTable, $maxRowsInSubtable = $maxRowsInTable,
* $columnToSortBy = Metrics::INDEX_NB_VISITS);
*
* $archiveProcessor->insertBlobRecords('MyPlugin_myFancyReport', $serializedData);
* }
*
* **Aggregating archive data**
*
* // function in Archiver descendant
* public function aggregateMultipleReports()
* {
* $archiveProcessor = $this->getProcessor();
*
* // aggregate a metric
* $archiveProcessor->aggregateNumericMetrics('MyPlugin_myFancyMetric');
* $archiveProcessor->aggregateNumericMetrics('MyPlugin_mySuperFancyMetric', 'max');
*
* // aggregate a report
* $archiveProcessor->aggregateDataTableRecords('MyPlugin_myFancyReport');
* }
*
*/
class ArchiveProcessor
{
/**
* @var \Piwik\DataAccess\ArchiveWriter
*/
private $archiveWriter;
/**
* @var \Piwik\DataAccess\LogAggregator
*/
private $logAggregator;
/**
* @var Archive
*/
public $archive = null;
/**
* @var Parameters
*/
private $params;
/**
* @var int
*/
private $numberOfVisits = false;
private $numberOfVisitsConverted = false;
public function __construct(Parameters $params, ArchiveWriter $archiveWriter, LogAggregator $logAggregator)
{
$this->params = $params;
$this->logAggregator = $logAggregator;
$this->archiveWriter = $archiveWriter;
}
protected function getArchive()
{
if (empty($this->archive)) {
$subPeriods = $this->params->getSubPeriods();
$idSites = $this->params->getIdSites();
$this->archive = Archive::factory($this->params->getSegment(), $subPeriods, $idSites);
/**
* @internal
*/
Piwik::postEvent('ArchiveProcessor.getArchive', [$this->archive]);
}
return $this->archive;
}
public function setNumberOfVisits($visits, $visitsConverted)
{
$this->numberOfVisits = $visits;
$this->numberOfVisitsConverted = $visitsConverted;
}
/**
* Returns the {@link Parameters} object containing the site, period and segment we're archiving
* data for.
*
* @return Parameters
* @api
*/
public function getParams()
{
return $this->params;
}
/**
* Returns a `{@link Piwik\DataAccess\LogAggregator}` instance for the site, period and segment this
* ArchiveProcessor will insert archive data for.
*
* @return LogAggregator
* @api
*/
public function getLogAggregator()
{
return $this->logAggregator;
}
/**
* Array of (column name before => column name renamed) of the columns for which sum operation is invalid.
* These columns will be renamed as per this mapping.
* @var array
*/
protected static $columnsToRenameAfterAggregation = array(
Metrics::INDEX_NB_UNIQ_VISITORS => Metrics::INDEX_SUM_DAILY_NB_UNIQ_VISITORS,
Metrics::INDEX_NB_USERS => Metrics::INDEX_SUM_DAILY_NB_USERS,
);
/**
* Sums records for every subperiod of the current period and inserts the result as the record
* for this period.
*
* DataTables are summed recursively so subtables will be summed as well.
*
* @param string|array $recordNames Name(s) of the report we are aggregating, eg, `'Referrers_type'`.
* @param int $maximumRowsInDataTableLevelZero Maximum number of rows allowed in the top level DataTable.
* @param int $maximumRowsInSubDataTable Maximum number of rows allowed in each subtable.
* @param string $columnToSortByBeforeTruncation The name of the column to sort by before truncating a DataTable.
* @param array $columnsAggregationOperation Operations for aggregating columns, see {@link Row::sumRow()}.
* @param array $columnsToRenameAfterAggregation Columns mapped to new names for columns that must change names
* when summed because they cannot be summed, eg,
* `array('nb_uniq_visitors' => 'sum_daily_nb_uniq_visitors')`.
* @param bool|array $countRowsRecursive if set to true, will calculate the recursive rows count for all record names
* which makes it slower. If you only need it for some records pass an array of
* recordNames that defines for which ones you need a recursive row count.
* @return array Returns the row counts of each aggregated report before truncation, eg,
*
* array(
* 'report1' => array('level0' => $report1->getRowsCount,
* 'recursive' => $report1->getRowsCountRecursive()),
* 'report2' => array('level0' => $report2->getRowsCount,
* 'recursive' => $report2->getRowsCountRecursive()),
* ...
* )
* @api
*/
public function aggregateDataTableRecords($recordNames,
$maximumRowsInDataTableLevelZero = null,
$maximumRowsInSubDataTable = null,
$columnToSortByBeforeTruncation = null,
&$columnsAggregationOperation = null,
$columnsToRenameAfterAggregation = null,
$countRowsRecursive = true)
{
/** @var LoggerInterface $logger */
$logger = StaticContainer::get(LoggerInterface::class);
if (!is_array($recordNames)) {
$recordNames = array($recordNames);
}
$archiveDescription = $this->params . '';
$nameToCount = array();
foreach ($recordNames as $recordName) {
$latestUsedTableId = Manager::getInstance()->getMostRecentTableId();
$logger->debug("aggregating record {record} [archive = {archive}]", [
'record' => $recordName,
'archive' => $archiveDescription,
]);
$table = $this->aggregateDataTableRecord($recordName, $columnsAggregationOperation, $columnsToRenameAfterAggregation);
$nameToCount[$recordName]['level0'] = $table->getRowsCount();
if ($countRowsRecursive === true || (is_array($countRowsRecursive) && in_array($recordName, $countRowsRecursive))) {
$nameToCount[$recordName]['recursive'] = $table->getRowsCountRecursive();
}
$blob = $table->getSerialized($maximumRowsInDataTableLevelZero, $maximumRowsInSubDataTable, $columnToSortByBeforeTruncation);
Common::destroy($table);
$this->insertBlobRecord($recordName, $blob);
unset($blob);
DataTable\Manager::getInstance()->deleteAll($latestUsedTableId);
}
return $nameToCount;
}
/**
* Aggregates one or more metrics for every subperiod of the current period and inserts the results
* as metrics for the current period.
*
* @param array|string $columns Array of metric names to aggregate.
* @param bool|string $operationToApply The operation to apply to the metric. Either `'sum'`, `'max'` or `'min'`.
* @return array|int Returns the array of aggregate values. If only one metric was aggregated,
* the aggregate value will be returned as is, not in an array.
* For example, if `array('nb_visits', 'nb_hits')` is supplied for `$columns`,
*
* array(
* 'nb_visits' => 3040,
* 'nb_hits' => 405
* )
*
* could be returned. If `array('nb_visits')` or `'nb_visits'` is used for `$columns`,
* then `3040` would be returned.
* @api
*/
public function aggregateNumericMetrics($columns, $operationToApply = false)
{
$metrics = $this->getAggregatedNumericMetrics($columns, $operationToApply);
foreach ($metrics as $column => $value) {
$this->insertNumericRecord($column, $value);
}
// if asked for only one field to sum
if (count($metrics) === 1) {
return reset($metrics);
}
// returns the array of records once summed
return $metrics;
}
public function getNumberOfVisits()
{
if ($this->numberOfVisits === false) {
throw new Exception("visits should have been set here");
}
return $this->numberOfVisits;
}
public function getNumberOfVisitsConverted()
{
return $this->numberOfVisitsConverted;
}
/**
* Caches multiple numeric records in the archive for this processor's site, period
* and segment.
*
* @param array $numericRecords A name-value mapping of numeric values that should be
* archived, eg,
*
* array('Referrers_distinctKeywords' => 23, 'Referrers_distinctCampaigns' => 234)
* @api
*/
public function insertNumericRecords($numericRecords)
{
foreach ($numericRecords as $name => $value) {
$this->insertNumericRecord($name, $value);
}
}
/**
* Caches a single numeric record in the archive for this processor's site, period and
* segment.
*
* Numeric values are not inserted if they equal `0`.
*
* @param string $name The name of the numeric value, eg, `'Referrers_distinctKeywords'`.
* @param float $value The numeric value.
* @api
*/
public function insertNumericRecord($name, $value)
{
$value = round($value ?? 0, 2);
$value = Common::forceDotAsSeparatorForDecimalPoint($value);
$this->archiveWriter->insertRecord($name, $value);
}
/**
* Caches one or more blob records in the archive for this processor's site, period
* and segment.
*
* @param string $name The name of the record, eg, 'Referrers_type'.
* @param string|array $values A blob string or an array of blob strings. If an array
* is used, the first element in the array will be inserted
* with the `$name` name. The others will be inserted with
* `$name . '_' . $index` as the record name (where $index is
* the index of the blob record in `$values`).
* @api
*/
public function insertBlobRecord($name, $values)
{
$this->archiveWriter->insertBlobRecord($name, $values);
}
/**
* This method selects all DataTables that have the name $name over the period.
* All these DataTables are then added together, and the resulting DataTable is returned.
*
* @param string $name
* @param array $columnsAggregationOperation Operations for aggregating columns, @see Row::sumRow()
* @param array $columnsToRenameAfterAggregation columns in the array (old name, new name) to be renamed as the sum operation is not valid on them (eg. nb_uniq_visitors->sum_daily_nb_uniq_visitors)
* @return DataTable
*/
protected function aggregateDataTableRecord($name, $columnsAggregationOperation = null, $columnsToRenameAfterAggregation = null)
{
try {
ErrorHandler::pushFatalErrorBreadcrumb(__CLASS__, ['name' => $name]);
// By default we shall aggregate all sub-tables.
$dataTableBlobs = $this->getArchive()->getBlob($name, Archive::ID_SUBTABLE_LOAD_ALL_SUBTABLES);
$dataTable = $this->getAggregatedDataTableMapFromBlobs($dataTableBlobs, $columnsAggregationOperation, $columnsToRenameAfterAggregation, $name);
} finally {
ErrorHandler::popFatalErrorBreadcrumb();
}
return $dataTable;
}
protected function getAggregatedDataTableMapFromBlobs(DataCollection $dataTableBlobs, $columnsAggregationOperation, $columnsToRenameAfterAggregation, $name)
{
$result = new DataTable();
if (!empty($columnsAggregationOperation)) {
$result->setMetadata(DataTable::COLUMN_AGGREGATION_OPS_METADATA_NAME, $columnsAggregationOperation);
}
$dataTableBlobs->forEachBlobExpanded(function ($reportBlobs, DataTableFactory $factory, $tableMetadata) use ($name, $result, $columnsToRenameAfterAggregation) {
$latestUsedTableId = Manager::getInstance()->getMostRecentTableId();
$toSum = $factory->make($reportBlobs, $index = [], $tableMetadata);
$latestUsedAfterCreatingToSum = Manager::getInstance()->getMostRecentTableId();
// see https://github.com/piwik/piwik/issues/4377
$toSum->filter(function ($table) use ($columnsToRenameAfterAggregation, $name) {
if ($this->areColumnsNotAlreadyRenamed($table)) {
/**
* This makes archiving and range dates a lot faster. Imagine we archive a week, then we will
* rename all columns of each 7 day archives. Afterwards we know the columns will be replaced in a
* week archive. When generating month archives, which uses mostly week archives, we do not have
* to replace those columns for the week archives again since we can be sure they were already
* replaced. Same when aggregating year and range archives. This can save up 10% or more when
* aggregating Month, Year and Range archives.
*/
$this->renameColumnsAfterAggregation($table, $columnsToRenameAfterAggregation);
}
});
$result->addDataTable($toSum);
DataTable\Manager::getInstance()->deleteAll($latestUsedTableId, $latestUsedAfterCreatingToSum);
});
return $result;
}
/**
* Note: public only for use in closure in PHP 5.3.
*
* @param $table
* @return \Piwik\Period
*/
public function areColumnsNotAlreadyRenamed($table)
{
$period = $table->getMetadata(DataTableFactory::TABLE_METADATA_PERIOD_INDEX);
return !$period || $period->getLabel() === 'day';
}
protected function getOperationForColumns($columns, $defaultOperation)
{
$operationForColumn = array();
foreach ($columns as $name) {
$operation = $defaultOperation;
if (empty($operation)) {
$operation = $this->guessOperationForColumn($name);
}
$operationForColumn[$name] = $operation;
}
return $operationForColumn;
}
protected function enrichWithUniqueVisitorsMetric(Row $row)
{
if ($row->getColumn('nb_uniq_visitors') === false
&& $row->getColumn('nb_users') === false
) {
return;
}
$periodLabel = $this->getParams()->getPeriod()->getLabel();
if (!SettingsPiwik::isUniqueVisitorsEnabled($periodLabel)) {
$row->deleteColumn('nb_uniq_visitors');
$row->deleteColumn('nb_users');
return;
}
$sites = $this->getIdSitesToComputeNbUniques();
if (count($sites) > 1 && Rules::shouldSkipUniqueVisitorsCalculationForMultipleSites()) {
if ($periodLabel != 'day') {
// for day we still keep the aggregated metric but for other periods we remove it as it becomes to
// inaccurate
$row->deleteColumn('nb_uniq_visitors');
$row->deleteColumn('nb_users');
}
return;
}
if (empty($sites)) {
// a plugin disabled running below query by removing all sites.
$row->deleteColumn('nb_uniq_visitors');
$row->deleteColumn('nb_users');
return;
}
if (count($sites) === 1) {
$uniqueVisitorsMetric = Metrics::INDEX_NB_UNIQ_VISITORS;
} else {
if (!SettingsPiwik::isSameFingerprintAcrossWebsites()) {
throw new Exception("Processing unique visitors across websites is enabled for this instance,
but to process this metric you must first set enable_fingerprinting_across_websites=1
in the config file, under the [Tracker] section.");
}
$uniqueVisitorsMetric = Metrics::INDEX_NB_UNIQ_FINGERPRINTS;
}
$metrics = array(
Metrics::INDEX_NB_USERS,
$uniqueVisitorsMetric
);
$uniques = $this->computeNbUniques($metrics, $sites);
// see edge case as described in https://github.com/piwik/piwik/issues/9357 where uniq_visitors might be higher
// than visits because we archive / process it after nb_visits. Between archiving nb_visits and nb_uniq_visitors
// there could have been a new visit leading to a higher nb_unique_visitors than nb_visits which is not possible
// by definition. In this case we simply use the visits metric instead of unique visitors metric.
$visits = $row->getColumn('nb_visits');
if ($visits !== false && $uniques[$uniqueVisitorsMetric] !== false) {
$uniques[$uniqueVisitorsMetric] = min($uniques[$uniqueVisitorsMetric], $visits);
}
$row->setColumn('nb_uniq_visitors', $uniques[$uniqueVisitorsMetric]);
$row->setColumn('nb_users', $uniques[Metrics::INDEX_NB_USERS]);
}
protected function guessOperationForColumn($column)
{
if (strpos($column, 'max_') === 0) {
return 'max';
}
if (strpos($column, 'min_') === 0) {
return 'min';
}
return 'sum';
}
private function getIdSitesToComputeNbUniques()
{
$params = $this->getParams();
$sites = array($params->getSite()->getId());
/**
* Triggered to change which site ids should be looked at when processing unique visitors and users.
*
* @param array &$idSites An array with one idSite. This site is being archived currently. To cancel the query
* you can change this value to an empty array. To include other sites in the query you
* can add more idSites to this list of idSites.
* @param Period $period The period that is being requested to be archived.
* @param Segment $segment The segment that is request to be archived.
*/
Piwik::postEvent('ArchiveProcessor.ComputeNbUniques.getIdSites', array(&$sites, $params->getPeriod(), $params->getSegment()));
return $sites;
}
/**
* Processes number of unique visitors for the given period
*
* This is the only Period metric (ie. week/month/year/range) that we process from the logs directly,
* since unique visitors cannot be summed like other metrics.
*
* @param array $metrics Metrics Ids for which to aggregates count of values
* @param int[] $sites A list of idSites that should be included
* @return array|null An array of metrics, where the key is metricid and the value is the metric value or null if
* the query was cancelled and not executed.
*/
protected function computeNbUniques($metrics, $sites)
{
$logAggregator = $this->getLogAggregator();
$sitesBackup = $logAggregator->getSites();
$logAggregator->setSites($sites);
try {
$query = $logAggregator->queryVisitsByDimension(array(), false, array(), $metrics);
} finally {
$logAggregator->setSites($sitesBackup);
}
$data = $query->fetch();
return $data;
}
/**
* If the DataTable is a Map, sums all DataTable in the map and return the DataTable.
*
*
* @param $data DataTable|DataTable\Map
* @param $columnsToRenameAfterAggregation array
* @return DataTable
*/
protected function getAggregatedDataTableMap($data, $columnsAggregationOperation)
{
$table = new DataTable();
if (!empty($columnsAggregationOperation)) {
$table->setMetadata(DataTable::COLUMN_AGGREGATION_OPS_METADATA_NAME, $columnsAggregationOperation);
}
if ($data instanceof DataTable\Map) {
// as $date => $tableToSum
$this->aggregatedDataTableMapsAsOne($data, $table);
} else {
$table->addDataTable($data);
}
return $table;
}
/**
* Aggregates the DataTable\Map into the destination $aggregated
* @param $map
* @param $aggregated
*/
protected function aggregatedDataTableMapsAsOne(Map $map, DataTable $aggregated)
{
foreach ($map->getDataTables() as $tableToAggregate) {
if ($tableToAggregate instanceof Map) {
$this->aggregatedDataTableMapsAsOne($tableToAggregate, $aggregated);
} else {
$aggregated->addDataTable($tableToAggregate);
}
}
}
/**
* Note: public only for use in closure in PHP 5.3.
*/
public function renameColumnsAfterAggregation(DataTable $table, $columnsToRenameAfterAggregation = null)
{
// Rename columns after aggregation
if (is_null($columnsToRenameAfterAggregation)) {
$columnsToRenameAfterAggregation = self::$columnsToRenameAfterAggregation;
}
if (empty($columnsToRenameAfterAggregation)) {
return;
}
foreach ($table->getRows() as $row) {
foreach ($columnsToRenameAfterAggregation as $oldName => $newName) {
$row->renameColumn($oldName, $newName);
}
$subTable = $row->getSubtable();
if ($subTable) {
$this->renameColumnsAfterAggregation($subTable, $columnsToRenameAfterAggregation);
}
}
}
protected function getAggregatedNumericMetrics($columns, $operationToApply)
{
if (!is_array($columns)) {
$columns = array($columns);
}
$operationForColumn = $this->getOperationForColumns($columns, $operationToApply);
$dataTable = $this->getArchive()->getDataTableFromNumeric($columns);
$results = $this->getAggregatedDataTableMap($dataTable, $operationForColumn);
if ($results->getRowsCount() > 1) {
throw new Exception("A DataTable is an unexpected state:" . var_export($results, true));
}
$rowMetrics = $results->getFirstRow();
if ($rowMetrics === false) {
$rowMetrics = new Row;
}
$this->enrichWithUniqueVisitorsMetric($rowMetrics);
$this->renameColumnsAfterAggregation($results, self::$columnsToRenameAfterAggregation);
$metrics = $rowMetrics->getColumns();
foreach ($columns as $name) {
if (!isset($metrics[$name])) {
$metrics[$name] = 0;
}
}
return $metrics;
}
/**
* Initiate archiving for a plugin during an ongoing archiving. The plugin can be another
* plugin or the same plugin.
*
* This method should be called during archiving when one plugin uses the report of another
* plugin with a segment. It will ensure reports for that segment & plugin will be archived
* without initiating archiving for every plugin with that segment (which would be a performance
* killer).
*
* @param string $plugin
* @param string $segment
*/
public function processDependentArchive($plugin, $segment)
{
$params = $this->getParams();
if (!$params->isRootArchiveRequest()) { // prevent all recursion
return;
}
$idSites = [$params->getSite()->getId()];
// important to use the original segment string when combining. As the API itself would combine the original string.
// this prevents a bug where the API would use the segment
// userId!@%2540matomo.org;userId!=hello%2540matomo.org;visitorType==new
// vs here we would use
// userId!@%40matomo.org;userId!=hello%40matomo.org;visitorType==new
// thus these would result in different segment hashes and therefore the reports would either show 0 or archive the data twice
$originSegmentString = $params->getSegment()->getOriginalString();
$newSegment = Segment::combine($originSegmentString, SegmentExpression::AND_DELIMITER, $segment);
if (!empty($originSegmentString) && $newSegment === $segment && $params->getRequestedPlugin() === $plugin) { // being processed now
return;
}
$newSegment = new Segment($newSegment, $idSites, $params->getDateStart(), $params->getDateEnd());
if (ArchiveProcessor\Rules::isSegmentPreProcessed($idSites, $newSegment)) {
// will be processed anyway
return;
}
$parameters = new ArchiveProcessor\Parameters($params->getSite(), $params->getPeriod(), $newSegment);
$parameters->onlyArchiveRequestedPlugin();
$parameters->setIsRootArchiveRequest(false);
$archiveLoader = new ArchiveProcessor\Loader($parameters);
$archiveLoader->prepareArchive($plugin);
}
public function getArchiveWriter()
{
return $this->archiveWriter;
}
}

124
matomo/core/ArchiveProcessor/ArchivingStatus.php Normal file
View file

@ -0,0 +1,124 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\ArchiveProcessor;
use Piwik\Concurrency\Lock;
use Piwik\Concurrency\LockBackend;
use Piwik\Container\StaticContainer;
class ArchivingStatus
{
const LOCK_KEY_PREFIX = 'Archiving';
const DEFAULT_ARCHIVING_TTL = 7200; // 2 hours
/**
* @var LockBackend
*/
private $lockBackend;
/**
* @var int
*/
private $archivingTTLSecs;
/**
* @var Lock[]
*/
private $lockStack = [];
public function __construct(LockBackend $lockBackend, $archivingTTLSecs = self::DEFAULT_ARCHIVING_TTL)
{
$this->lockBackend = $lockBackend;
$this->archivingTTLSecs = $archivingTTLSecs;
}
public function archiveStarted(Parameters $params)
{
$lock = $this->makeArchivingLock($params);
$locked = $lock->acquireLock('', $this->archivingTTLSecs);
if ($locked) {
array_push($this->lockStack, $lock);
}
return $locked;
}
/**
* Try to acquire the lock that is acquired before starting archiving. If it is acquired, it
* means archiving is not ongoing. If it is not acquired, then archiving is ongoing.
*
* @param Parameters $params
* @param $doneFlag
* @return Lock
*/
public function acquireArchiveInProgressLock($idSite, $date1, $date2, $period, $doneFlag)
{
$lock = $this->makeArchivingLockFromDoneFlag($idSite, $date1, $date2, $period, $doneFlag);
$lock->acquireLock('', $ttl = 1);
return $lock;
}
public function archiveFinished()
{
$lock = array_pop($this->lockStack);
$lock->unlock();
}
public function getCurrentArchivingLock()
{
if (empty($this->lockStack)) {
return null;
}
return end($this->lockStack);
}
public function getSitesCurrentlyArchiving()
{
$lockMeta = new Lock($this->lockBackend, self::LOCK_KEY_PREFIX . '.');
$acquiredLocks = $lockMeta->getAllAcquiredLockKeys();
$sitesCurrentlyArchiving = [];
foreach ($acquiredLocks as $lockKey) {
$parts = explode('.', $lockKey);
if (!isset($parts[1])) {
continue;
}
$sitesCurrentlyArchiving[] = (int) $parts[1];
}
$sitesCurrentlyArchiving = array_unique($sitesCurrentlyArchiving);
$sitesCurrentlyArchiving = array_values($sitesCurrentlyArchiving);
return $sitesCurrentlyArchiving;
}
/**
* @return Lock
*/
private function makeArchivingLock(Parameters $params)
{
$doneFlag = Rules::getDoneStringFlagFor([$params->getSite()->getId()], $params->getSegment(),
$params->getPeriod()->getLabel(), $params->getRequestedPlugin());
return $this->makeArchivingLockFromDoneFlag($params->getSite()->getId(), $params->getSite()->getId(), $params->getPeriod()->getDateStart()->toString(),
$params->getPeriod()->getDateEnd()->toString(), $doneFlag);
}
private function makeArchivingLockFromDoneFlag($idSite, $date1, $date2, $period, $doneFlag)
{
$lockKeyParts = [
self::LOCK_KEY_PREFIX,
$idSite,
// md5 to keep it within the 70 char limit in the table
md5($period . $date1 . ',' . $date2 . $doneFlag),
];
$lockKeyPrefix = implode('.', $lockKeyParts);
return new Lock(StaticContainer::get(LockBackend::class), $lockKeyPrefix, $this->archivingTTLSecs);
}
}

586
matomo/core/ArchiveProcessor/Loader.php Normal file
View file

@ -0,0 +1,586 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\ArchiveProcessor;
use Piwik\Archive\ArchiveInvalidator;
use Piwik\Cache;
use Piwik\Common;
use Piwik\Config;
use Piwik\Container\StaticContainer;
use Piwik\Context;
use Piwik\DataAccess\ArchiveSelector;
use Piwik\DataAccess\ArchiveWriter;
use Piwik\DataAccess\Model;
use Piwik\DataAccess\RawLogDao;
use Piwik\Date;
use Piwik\Period;
use Piwik\Piwik;
use Piwik\SettingsServer;
use Piwik\Site;
use Psr\Log\LoggerInterface;
use Piwik\CronArchive\SegmentArchiving;
/**
* This class uses PluginsArchiver class to trigger data aggregation and create archives.
*/
class Loader
{
private static $archivingDepth = 0;
/**
* @var Parameters
*/
protected $params;
/**
* @var ArchiveInvalidator
*/
private $invalidator;
/**
* @var \Matomo\Cache\Cache
*/
private $cache;
/**
* @var LoggerInterface
*/
private $logger;
/**
* @var RawLogDao
*/
private $rawLogDao;
/**
* @var Model
*/
private $dataAccessModel;
public function __construct(Parameters $params, $invalidateBeforeArchiving = false)
{
$this->params = $params;
$this->invalidateBeforeArchiving = $invalidateBeforeArchiving;
$this->invalidator = StaticContainer::get(ArchiveInvalidator::class);
$this->cache = Cache::getTransientCache();
$this->logger = StaticContainer::get(LoggerInterface::class);
$this->rawLogDao = new RawLogDao();
$this->dataAccessModel = new Model();
}
/**
* @return bool
*/
protected function isThereSomeVisits($visits)
{
return $visits > 0;
}
/**
* @return bool
*/
protected function mustProcessVisitCount($visits)
{
return $visits === false;
}
public function prepareArchive($pluginName)
{
return Context::changeIdSite($this->params->getSite()->getId(), function () use ($pluginName) {
try {
++self::$archivingDepth;
return $this->prepareArchiveImpl($pluginName);
} finally {
--self::$archivingDepth;
}
});
}
/**
* @throws \Exception
*/
private function prepareArchiveImpl($pluginName)
{
$this->params->setRequestedPlugin($pluginName);
if (SettingsServer::isArchivePhpTriggered()) {
$requestedReport = Common::getRequestVar('requestedReport', '', 'string');
if (!empty($requestedReport)) {
$this->params->setArchiveOnlyReport($requestedReport);
}
}
// invalidate existing archives before we start archiving in case data was tracked in the past. if the archive is
// made invalid, we will correctly re-archive below.
if ($this->invalidateBeforeArchiving
&& Rules::isBrowserTriggerEnabled()
) {
$this->invalidatedReportsIfNeeded();
}
// load existing data from archive
$data = $this->loadArchiveData();
if (sizeof($data) == 2) {
return $data;
}
list($idArchives, $visits, $visitsConverted) = $data;
// only lock meet those conditions
if ($this->params->isRootArchiveRequest() && !SettingsServer::isArchivePhpTriggered()) {
$lockId = $this->makeArchivingLockId();
//ini lock
$lock = new LoaderLock($lockId);
//set mysql lock the entire process if another process is running
$lock->setLock();
try {
$data = $this->loadArchiveData();
if (sizeof($data) == 2) {
return $data;
}
list($idArchives, $visits, $visitsConverted) = $data;
return $this->insertArchiveData($visits, $visitsConverted);
} finally {
$lock->unlock();
}
} else {
return $this->insertArchiveData($visits, $visitsConverted);
}
}
/**
* @param $visits
* @param $visitsConverted
* @return array|false[]
*/
protected function insertArchiveData($visits, $visitsConverted)
{
if (SettingsServer::isArchivePhpTriggered()) {
$this->logger->info("initiating archiving via core:archive for " . $this->params);
}
list($visits, $visitsConverted) = $this->prepareCoreMetricsArchive($visits, $visitsConverted);
list($idArchive, $visits) = $this->prepareAllPluginsArchive($visits, $visitsConverted);
if ($this->isThereSomeVisits($visits) || PluginsArchiver::doesAnyPluginArchiveWithoutVisits()) {
return [[$idArchive], $visits];
}
return [false, false];
}
/**
* @return string
* @throws \Exception
*/
private function makeArchivingLockId()
{
$doneFlag = Rules::getDoneStringFlagFor([$this->params->getSite()->getId()], $this->params->getSegment(),
$this->params->getPeriod()->getLabel(), $this->params->getRequestedPlugin());
return $this->params->getPeriod()->getDateStart()->toString() . $this->params->getPeriod()->getDateEnd()->toString() .'.'. $doneFlag;
}
/**
* @return array|false[]
*/
protected function loadArchiveData()
{
// this hack was used to check the main function goes to return or continue
// NOTE: $idArchives will contain the latest DONE_OK/DONE_INVALIDATED archive as well as any partial archives
// with a ts_archived >= the DONE_OK/DONE_INVALIDATED date.
list($idArchives, $visits, $visitsConverted, $isAnyArchiveExists, $tsArchived, $value) = $this->loadExistingArchiveIdFromDb();
if (!empty($idArchives)
&& !Rules::isActuallyForceArchivingSinglePlugin()
&& !$this->shouldForceInvalidatedArchive($value, $tsArchived)) {
// we have a usable idarchive (it's not invalidated and it's new enough), and we are not archiving
// a single report
return [$idArchives, $visits];
}
// NOTE: this optimization helps when archiving large periods. eg, if archiving a year w/ a segment where
// there are not visits in the entire year, we don't have to go through and do anything. but, w/o this
// code, we will end up launching archiving for each month, week and day, even though we don't have to.
//
// we don't create an archive in this case, because the archive may be in progress in some way, so a 0
// visits archive can be inaccurate in the long run.
if ($this->canSkipThisArchive()) {
if (!empty($idArchives)) {
return [$idArchives, $visits];
} else {
return [false, 0];
}
}
return [$idArchives, $visits, $visitsConverted];
}
/**
* Prepares the core metrics if needed.
*
* @param $visits
* @return array
*/
protected function prepareCoreMetricsArchive($visits, $visitsConverted)
{
$createSeparateArchiveForCoreMetrics = $this->mustProcessVisitCount($visits)
&& !$this->doesRequestedPluginIncludeVisitsSummary();
if ($createSeparateArchiveForCoreMetrics) {
$requestedPlugin = $this->params->getRequestedPlugin();
$requestedReport = $this->params->getArchiveOnlyReport();
$isPartialArchive = $this->params->isPartialArchive();
$this->params->setRequestedPlugin('VisitsSummary');
$this->params->setArchiveOnlyReport(null);
$this->params->setIsPartialArchive(false);
$metrics = Context::executeWithQueryParameters(['requestedReport' => ''], function () {
$pluginsArchiver = new PluginsArchiver($this->params);
$metrics = $pluginsArchiver->callAggregateCoreMetrics();
$pluginsArchiver->finalizeArchive();
return $metrics;
});
$this->params->setRequestedPlugin($requestedPlugin);
$this->params->setArchiveOnlyReport($requestedReport);
$this->params->setIsPartialArchive($isPartialArchive);
$visits = $metrics['nb_visits'];
$visitsConverted = $metrics['nb_visits_converted'];
}
return array($visits, $visitsConverted);
}
protected function prepareAllPluginsArchive($visits, $visitsConverted)
{
$pluginsArchiver = new PluginsArchiver($this->params);
if ($this->mustProcessVisitCount($visits)
|| $this->doesRequestedPluginIncludeVisitsSummary()
) {
$metrics = $pluginsArchiver->callAggregateCoreMetrics();
$visits = $metrics['nb_visits'];
$visitsConverted = $metrics['nb_visits_converted'];
}
$forceArchivingWithoutVisits = !$this->isThereSomeVisits($visits) && $this->shouldArchiveForSiteEvenWhenNoVisits();
$pluginsArchiver->callAggregateAllPlugins($visits, $visitsConverted, $forceArchivingWithoutVisits);
$idArchive = $pluginsArchiver->finalizeArchive();
return array($idArchive, $visits);
}
protected function doesRequestedPluginIncludeVisitsSummary()
{
$processAllReportsIncludingVisitsSummary =
Rules::shouldProcessReportsAllPlugins(array($this->params->getSite()->getId()), $this->params->getSegment(), $this->params->getPeriod()->getLabel());
$doesRequestedPluginIncludeVisitsSummary = $processAllReportsIncludingVisitsSummary
|| $this->params->getRequestedPlugin() == 'VisitsSummary';
return $doesRequestedPluginIncludeVisitsSummary;
}
protected function isArchivingForcedToTrigger()
{
$period = $this->params->getPeriod()->getLabel();
$debugSetting = 'always_archive_data_period'; // default
if ($period == 'day') {
$debugSetting = 'always_archive_data_day';
} elseif ($period == 'range') {
$debugSetting = 'always_archive_data_range';
}
return (bool) Config::getInstance()->Debug[$debugSetting];
}
/**
* Returns the idArchive if the archive is available in the database for the requested plugin.
* Returns false if the archive needs to be processed.
*
* (public for tests)
*
* @return array
*/
public function loadExistingArchiveIdFromDb()
{
if ($this->isArchivingForcedToTrigger()) {
$this->logger->debug("Archiving forced to trigger for {$this->params}.");
// return no usable archive found, and no existing archive. this will skip invalidation, which should
// be fine since we just force archiving.
return [false, false, false, false, false, false];
}
$minDatetimeArchiveProcessedUTC = $this->getMinTimeArchiveProcessed();
$result = ArchiveSelector::getArchiveIdAndVisits($this->params, $minDatetimeArchiveProcessedUTC);
return $result;
}
/**
* Returns the minimum archive processed datetime to look at. Only public for tests.
*
* @return int|bool Datetime timestamp, or false if must look at any archive available
*/
protected function getMinTimeArchiveProcessed()
{
// for range periods we can archive in a browser request request, make sure to check for the ttl no matter what
$isRangeArchiveAndArchivingEnabled = $this->params->getPeriod()->getLabel() == 'range'
&& Rules::isArchivingEnabledFor([$this->params->getSite()->getId()], $this->params->getSegment(), $this->params->getPeriod()->getLabel());
if (!$isRangeArchiveAndArchivingEnabled) {
$endDateTimestamp = self::determineIfArchivePermanent($this->params->getDateEnd());
if ($endDateTimestamp) {
// past archive
return $endDateTimestamp;
}
}
$dateStart = $this->params->getDateStart();
$period = $this->params->getPeriod();
$segment = $this->params->getSegment();
$site = $this->params->getSite();
// in-progress archive
return Rules::getMinTimeProcessedForInProgressArchive($dateStart, $period, $segment, $site);
}
protected static function determineIfArchivePermanent(Date $dateEnd)
{
$now = time();
$endTimestampUTC = strtotime($dateEnd->getDateEndUTC());
if ($endTimestampUTC <= $now) {
// - if the period we are looking for is finished, we look for a ts_archived that
// is greater than the last day of the archive
return $endTimestampUTC;
}
return false;
}
private function shouldArchiveForSiteEvenWhenNoVisits()
{
$idSitesToArchive = $this->getIdSitesToArchiveWhenNoVisits();
return in_array($this->params->getSite()->getId(), $idSitesToArchive);
}
private function getIdSitesToArchiveWhenNoVisits()
{
$cache = Cache::getTransientCache();
$cacheKey = 'Archiving.getIdSitesToArchiveWhenNoVisits';
if (!$cache->contains($cacheKey)) {
$idSites = array();
// leaving undocumented unless decided otherwise
Piwik::postEvent('Archiving.getIdSitesToArchiveWhenNoVisits', array(&$idSites));
$cache->save($cacheKey, $idSites);
}
return $cache->fetch($cacheKey);
}
// public for tests
public function getReportsToInvalidate()
{
$sitesPerDays = $this->invalidator->getRememberedArchivedReportsThatShouldBeInvalidated();
foreach ($sitesPerDays as $dateStr => $siteIds) {
if (empty($siteIds)
|| !in_array($this->params->getSite()->getId(), $siteIds)
) {
unset($sitesPerDays[$dateStr]);
}
$date = Date::factory($dateStr);
if ($date->isEarlier($this->params->getPeriod()->getDateStart())
|| $date->isLater($this->params->getPeriod()->getDateEnd())
) { // date in list is not the current date, so ignore it
unset($sitesPerDays[$dateStr]);
}
}
return $sitesPerDays;
}
private function invalidatedReportsIfNeeded()
{
$sitesPerDays = $this->getReportsToInvalidate();
if (empty($sitesPerDays)) {
return;
}
foreach ($sitesPerDays as $date => $siteIds) {
try {
$this->invalidator->markArchivesAsInvalidated([$this->params->getSite()->getId()], array(Date::factory($date)), false, $this->params->getSegment());
} catch (\Exception $e) {
Site::clearCache();
throw $e;
}
}
Site::clearCache();
}
public function canSkipThisArchive()
{
$params = $this->params;
$idSite = $params->getSite()->getId();
$isWebsiteUsingTracker = $this->isWebsiteUsingTheTracker($idSite);
$isArchivingForcedWhenNoVisits = $this->shouldArchiveForSiteEvenWhenNoVisits();
$hasSiteVisitsBetweenTimeframe = $this->hasSiteVisitsBetweenTimeframe($idSite, $params->getPeriod());
$hasChildArchivesInPeriod = $this->dataAccessModel->hasChildArchivesInPeriod($idSite, $params->getPeriod());
if ($this->canSkipArchiveForSegment()) {
return true;
}
return $isWebsiteUsingTracker
&& !$isArchivingForcedWhenNoVisits
&& !$hasSiteVisitsBetweenTimeframe
&& !$hasChildArchivesInPeriod;
}
public function canSkipArchiveForSegment()
{
$params = $this->params;
if ($params->getSegment()->isEmpty()) {
return false;
}
if (!empty($params->getRequestedPlugin()) && Rules::isSegmentPluginArchivingDisabled($params->getRequestedPlugin(), $params->getSite()->getId())) {
return true;
}
/** @var SegmentArchiving */
$segmentArchiving = StaticContainer::get(SegmentArchiving::class);
$segmentInfo = $segmentArchiving->findSegmentForHash($params->getSegment()->getHash(), $params->getSite()->getId());
if (!$segmentInfo) {
return false;
}
$segmentArchiveStartDate = $segmentArchiving->getReArchiveSegmentStartDate($segmentInfo);
if ($segmentArchiveStartDate !==null && $segmentArchiveStartDate->isLater($params->getPeriod()->getDateEnd()->getEndOfDay())) {
$doneFlag = Rules::getDoneStringFlagFor(
[$params->getSite()->getId()],
$params->getSegment(),
$params->getPeriod()->getLabel(),
$params->getRequestedPlugin()
);
// if there is no invalidation where the report is null, we can skip
// if we have invalidations for the period and name, but only for a specific reports, we can skip
// if the report is not null we only want to rearchive if we have invalidation for that report
// if we don't find invalidation for that report, we can skip
return !$this->dataAccessModel->hasInvalidationForPeriodAndName($params->getSite()->getId(), $params->getPeriod(), $doneFlag, $params->getArchiveOnlyReport());
}
return false;
}
private function isWebsiteUsingTheTracker($idSite)
{
$idSitesNotUsingTracker = self::getSitesNotUsingTracker();
$isUsingTracker = !in_array($idSite, $idSitesNotUsingTracker);
return $isUsingTracker;
}
public static function getSitesNotUsingTracker()
{
$cache = Cache::getTransientCache();
$cacheKey = 'Archiving.isWebsiteUsingTheTracker';
$idSitesNotUsingTracker = $cache->fetch($cacheKey);
if ($idSitesNotUsingTracker === false || !isset($idSitesNotUsingTracker)) {
// we want to trigger event only once
$idSitesNotUsingTracker = array();
/**
* This event is triggered when detecting whether there are sites that do not use the tracker.
*
* By default we only archive a site when there was actually any visit since the last archiving.
* However, some plugins do import data from another source instead of using the tracker and therefore
* will never have any visits for this site. To make sure we still archive data for such a site when
* archiving for this site is requested, you can listen to this event and add the idSite to the list of
* sites that do not use the tracker.
*
* @param bool $idSitesNotUsingTracker The list of idSites that rather import data instead of using the tracker
*/
Piwik::postEvent('CronArchive.getIdSitesNotUsingTracker', array(&$idSitesNotUsingTracker));
$cache->save($cacheKey, $idSitesNotUsingTracker);
}
return $idSitesNotUsingTracker;
}
private function hasSiteVisitsBetweenTimeframe($idSite, Period $period)
{
$timezone = Site::getTimezoneFor($idSite);
list($date1, $date2) = $period->getBoundsInTimezone($timezone);
return $this->rawLogDao->hasSiteVisitsBetweenTimeframe($date1->getDatetime(), $date2->getDatetime(), $idSite);
}
public static function getArchivingDepth()
{
return self::$archivingDepth;
}
private function shouldForceInvalidatedArchive($value, $tsArchived)
{
$params = $this->params;
// the archive is invalidated and we are in a browser request that is allowed archive it
if ($value == ArchiveWriter::DONE_INVALIDATED
&& Rules::isArchivingEnabledFor([$params->getSite()->getId()], $params->getSegment(), $params->getPeriod()->getLabel())
) {
// if coming from core:archive, force rearchiving, since if we don't the entry will be removed from archive_invalidations
// w/o being rearchived
if (SettingsServer::isArchivePhpTriggered()) {
return true;
}
// if coming from a browser request, and period does not contain today, force rearchiving
$timezone = $params->getSite()->getTimezone();
if (!$params->getPeriod()->isDateInPeriod(Date::factoryInTimezone('today', $timezone))) {
return true;
}
// if coming from a browser request, and period does contain today, check the ttl for the period (done just below this)
$minDatetimeArchiveProcessedUTC = Rules::getMinTimeProcessedForInProgressArchive(
$params->getDateStart(), $params->getPeriod(), $params->getSegment(), $params->getSite());
$minDatetimeArchiveProcessedUTC = Date::factory($minDatetimeArchiveProcessedUTC);
if ($minDatetimeArchiveProcessedUTC
&& Date::factory($tsArchived)->isEarlier($minDatetimeArchiveProcessedUTC)
) {
return true;
}
}
return false;
}
}

65
matomo/core/ArchiveProcessor/LoaderLock.php Normal file
View file

@ -0,0 +1,65 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\ArchiveProcessor;
use Piwik\Db;
use Piwik\SettingsPiwik;
class LoaderLock
{
const MAX_LEN_LOCK_NAME = 64;
const MAX_LOCK_TIME = 60; //in seconds
protected $id;
/**
* @param string $id
* @throws \Exception
*/
public function __construct($id)
{
// instanceId is needed for multi tenant database solution
$id = SettingsPiwik::getPiwikInstanceId() . $id;
if (mb_strlen($id) >= self::MAX_LEN_LOCK_NAME) {
//convert ot prefix and md5 full length
$id = mb_substr($id, 0, 32) . md5($id);
}
$this->id = $id;
}
public function setLock()
{
Db::fetchOne('SELECT GET_LOCK(?,?)', array($this->id, self::MAX_LOCK_TIME));
}
public function unLock()
{
Db::query('DO RELEASE_LOCK(?)', array($this->id));
}
public function getId()
{
return $this->id;
}
/**
* @description check if the lock is available to user
* @param string $key
* @return bool
* @throws \Exception
*/
public static function isLockAvailable($key)
{
return (bool)Db::fetchOne('SELECT IS_FREE_LOCK(?)', [$key]);
}
}

326
matomo/core/ArchiveProcessor/Parameters.php Normal file
View file

@ -0,0 +1,326 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\ArchiveProcessor;
use Piwik\Date;
use Piwik\Log;
use Piwik\Period;
use Piwik\Piwik;
use Piwik\Segment;
use Piwik\Site;
/**
* Contains the analytics parameters for the reports that are currently being archived. The analytics
* parameters include the **website** the reports describe, the **period** of time the reports describe
* and the **segment** used to limit the visit set.
*/
class Parameters
{
/**
* @var Site
*/
private $site = null;
/**
* @var Period
*/
private $period = null;
/**
* @var Segment
*/
private $segment = null;
/**
* @var string Plugin name which triggered this archive processor
*/
private $requestedPlugin = false;
private $onlyArchiveRequestedPlugin = false;
/**
* @var bool
*/
private $isRootArchiveRequest = true;
/**
* @var string
*/
private $archiveOnlyReport = null;
/**
* @var bool
*/
private $isArchiveOnlyReportHandled;
/**
* Constructor.
*
* @ignore
*/
public function __construct(Site $site, Period $period, Segment $segment)
{
$this->site = $site;
$this->period = $period;
$this->segment = $segment;
}
/**
* If we want to archive only a single report, we can request that via this method.
* It is up to each plugin's archiver to respect the setting.
*
* @param string $archiveOnlyReport
* @api
*/
public function setArchiveOnlyReport($archiveOnlyReport)
{
$this->archiveOnlyReport = $archiveOnlyReport;
}
/**
* Gets the report we want to archive specifically, or null if none was specified.
*
* @return string|null
* @api
*/
public function getArchiveOnlyReport()
{
return $this->archiveOnlyReport;
}
/**
* @ignore
*/
public function setRequestedPlugin($plugin)
{
$this->requestedPlugin = $plugin;
}
/**
* @ignore
*/
public function onlyArchiveRequestedPlugin()
{
$this->onlyArchiveRequestedPlugin = true;
}
/**
* @ignore
*/
public function shouldOnlyArchiveRequestedPlugin()
{
return $this->onlyArchiveRequestedPlugin;
}
/**
* @ignore
*/
public function getRequestedPlugin()
{
return $this->requestedPlugin;
}
/**
* Returns the period we are computing statistics for.
*
* @return Period
* @api
*/
public function getPeriod()
{
return $this->period;
}
/**
* Returns the array of Period which make up this archive.
*
* @return \Piwik\Period[]
* @ignore
*/
public function getSubPeriods()
{
if ($this->getPeriod()->getLabel() == 'day') {
return array( $this->getPeriod() );
}
return $this->getPeriod()->getSubperiods();
}
/**
* @return array
* @ignore
*/
public function getIdSites()
{
$idSite = $this->getSite()->getId();
$idSites = array($idSite);
Piwik::postEvent('ArchiveProcessor.Parameters.getIdSites', array(&$idSites, $this->getPeriod()));
return $idSites;
}
/**
* Returns the site we are computing statistics for.
*
* @return Site
* @api
*/
public function getSite()
{
return $this->site;
}
/**
* The Segment used to limit the set of visits that are being aggregated.
*
* @return Segment
* @api
*/
public function getSegment()
{
return $this->segment;
}
/**
* Returns the end day of the period in the site's timezone.
*
* @return Date
*/
public function getDateEnd()
{
return $this->getPeriod()->getDateEnd()->setTimezone($this->getSite()->getTimezone());
}
/**
* Returns the start day of the period in the site's timezone.
*
* @return Date
*/
public function getDateStart()
{
return $this->getPeriod()->getDateStart()->setTimezone($this->getSite()->getTimezone());
}
/**
* Returns the start day of the period in the site's timezone (includes the time of day).
*
* @return Date
*/
public function getDateTimeStart()
{
return $this->getPeriod()->getDateTimeStart()->setTimezone($this->getSite()->getTimezone());
}
/**
* Returns the end day of the period in the site's timezone (includes the time of day).
*
* @return Date
*/
public function getDateTimeEnd()
{
return $this->getPeriod()->getDateTimeEnd()->setTimezone($this->getSite()->getTimezone());
}
/**
* @return bool
*/
public function isSingleSiteDayArchive()
{
return $this->isDayArchive() && $this->isSingleSite();
}
/**
* @return bool
*/
public function isDayArchive()
{
$period = $this->getPeriod();
$secondsInPeriod = $period->getDateEnd()->getTimestampUTC() - $period->getDateStart()->getTimestampUTC();
$oneDay = $secondsInPeriod < Date::NUM_SECONDS_IN_DAY;
return $oneDay;
}
public function isSingleSite()
{
return count($this->getIdSites()) == 1;
}
public function logStatusDebug()
{
$temporary = 'definitive archive';
Log::debug(
"%s archive, idSite = %d (%s), segment '%s', plugin = '%s', report = '%s', UTC datetime [%s -> %s]",
$this->getPeriod()->getLabel(),
$this->getSite()->getId(),
$temporary,
$this->getSegment()->getString(),
$this->getRequestedPlugin(),
$this->getArchiveOnlyReport(),
$this->getDateStart()->getDateStartUTC(),
$this->getDateEnd()->getDateEndUTC()
);
}
/**
* Returns `true` if these parameters are part of an initial archiving request.
* Returns `false` if these parameters are for an archiving request that was initiated
* during archiving.
*
* @return bool
*/
public function isRootArchiveRequest()
{
return $this->isRootArchiveRequest;
}
/**
* Sets whether these parameters are part of the initial archiving request or if they are
* for a request that was initiated during archiving.
*
* @param $isRootArchiveRequest
*/
public function setIsRootArchiveRequest($isRootArchiveRequest)
{
$this->isRootArchiveRequest = $isRootArchiveRequest;
}
public function __toString()
{
return "[idSite = {$this->getSite()->getId()}, period = {$this->getPeriod()->getLabel()} {$this->getPeriod()->getRangeString()}, segment = {$this->getSegment()->getString()}, plugin = {$this->getRequestedPlugin()}, report = {$this->getArchiveOnlyReport()}]";
}
/**
* Returns whether the setArchiveOnlyReport() was handled by an Archiver.
*
* @return bool
*/
public function isPartialArchive()
{
if (!$this->getRequestedPlugin()) { // sanity check, partial archives are only for single reports
return false;
}
return $this->isArchiveOnlyReportHandled;
}
/**
* If a plugin's archiver handles the setArchiveOnlyReport() setting, it should call this method
* so it is known that the archive only contains the requested report. This should be called
* in an Archiver's __construct method.
*
* @param bool $isArchiveOnlyReportHandled
*/
public function setIsPartialArchive($isArchiveOnlyReportHandled)
{
$this->isArchiveOnlyReportHandled = $isArchiveOnlyReportHandled;
}
}

338
matomo/core/ArchiveProcessor/PluginsArchiver.php Normal file
View file

@ -0,0 +1,338 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\ArchiveProcessor;
use Piwik\ArchiveProcessor;
use Piwik\Container\StaticContainer;
use Piwik\CronArchive\Performance\Logger;
use Piwik\DataAccess\ArchiveWriter;
use Piwik\DataAccess\LogAggregator;
use Piwik\DataTable\Manager;
use Piwik\Metrics;
use Piwik\Piwik;
use Piwik\Plugin\Archiver;
use Piwik\Log;
use Piwik\Timer;
use Exception;
/**
* This class creates the Archiver objects found in plugins and will trigger aggregation,
* so each plugin can process their reports.
*/
class PluginsArchiver
{
/**
* @var string|null
*/
private static $currentPluginBeingArchived = null;
/**
* @param ArchiveProcessor $archiveProcessor
*/
public $archiveProcessor;
/**
* @var Parameters
*/
protected $params;
/**
* @var LogAggregator
*/
private $logAggregator;
/**
* Public only for tests. Won't be necessary after DI changes are complete.
*
* @var Archiver[] $archivers
*/
public static $archivers = array();
/**
* Defines if we should aggregate from raw data by using MySQL queries (when true) or aggregate archives (when false)
* @var bool
*/
private $shouldAggregateFromRawData;
public function __construct(Parameters $params, ArchiveWriter $archiveWriter = null)
{
$this->params = $params;
$this->archiveWriter = $archiveWriter ?: new ArchiveWriter($this->params);
$this->archiveWriter->initNewArchive();
$this->logAggregator = new LogAggregator($params);
$this->logAggregator->allowUsageSegmentCache();
$this->archiveProcessor = new ArchiveProcessor($this->params, $this->archiveWriter, $this->logAggregator);
$shouldAggregateFromRawData = $this->params->isSingleSiteDayArchive();
/**
* Triggered to detect if the archiver should aggregate from raw data by using MySQL queries (when true)
* or by aggregate archives (when false). Typically, data is aggregated from raw data for "day" period, and
* aggregregated from archives for all other periods.
*
* @param bool $shouldAggregateFromRawData Set to true, to aggregate from raw data, or false to aggregate multiple reports.
* @param Parameters $params
*/
Piwik::postEvent('ArchiveProcessor.shouldAggregateFromRawData', array(&$shouldAggregateFromRawData, $this->params));
$this->shouldAggregateFromRawData = $shouldAggregateFromRawData;
}
/**
* If period is day, will get the core metrics (including visits) from the logs.
* If period is != day, will sum the core metrics from the existing archives.
* @return array Core metrics
*/
public function callAggregateCoreMetrics()
{
$this->logAggregator->cleanup();
$this->logAggregator->setQueryOriginHint('Core');
if ($this->shouldAggregateFromRawData) {
$metrics = $this->aggregateDayVisitsMetrics();
} else {
$metrics = $this->aggregateMultipleVisitsMetrics();
}
if (empty($metrics)) {
return array(
'nb_visits' => false,
'nb_visits_converted' => false
);
}
return array(
'nb_visits' => $metrics['nb_visits'],
'nb_visits_converted' => $metrics['nb_visits_converted']
);
}
/**
* Instantiates the Archiver class in each plugin that defines it,
* and triggers Aggregation processing on these plugins.
*/
public function callAggregateAllPlugins($visits, $visitsConverted, $forceArchivingWithoutVisits = false)
{
Log::debug("PluginsArchiver::%s: Initializing archiving process for all plugins [visits = %s, visits converted = %s]",
__FUNCTION__, $visits, $visitsConverted);
/** @var Logger $performanceLogger */
$performanceLogger = StaticContainer::get(Logger::class);
$this->archiveProcessor->setNumberOfVisits($visits, $visitsConverted);
$archivers = static::getPluginArchivers();
foreach ($archivers as $pluginName => $archiverClass) {
// We clean up below all tables created during this function call (and recursive calls)
$latestUsedTableId = Manager::getInstance()->getMostRecentTableId();
/** @var Archiver $archiver */
$archiver = $this->makeNewArchiverObject($archiverClass, $pluginName);
if (!$archiver->isEnabled()) {
Log::debug("PluginsArchiver::%s: Skipping archiving for plugin '%s' (disabled).", __FUNCTION__, $pluginName);
continue;
}
if (!$forceArchivingWithoutVisits && !$visits && !$archiver->shouldRunEvenWhenNoVisits()) {
Log::debug("PluginsArchiver::%s: Skipping archiving for plugin '%s' (no visits).", __FUNCTION__, $pluginName);
continue;
}
if ($this->shouldProcessReportsForPlugin($pluginName)) {
$this->logAggregator->setQueryOriginHint($pluginName);
try {
self::$currentPluginBeingArchived = $pluginName;
$period = $this->params->getPeriod()->getLabel();
$timer = new Timer();
if ($this->shouldAggregateFromRawData) {
Log::debug("PluginsArchiver::%s: Archiving $period reports for plugin '%s' from raw data.", __FUNCTION__, $pluginName);
$archiver->callAggregateDayReport();
} else {
Log::debug("PluginsArchiver::%s: Archiving $period reports for plugin '%s' using reports for smaller periods.", __FUNCTION__, $pluginName);
$archiver->callAggregateMultipleReports();
}
$this->logAggregator->setQueryOriginHint('');
$performanceLogger->logMeasurement('plugin', $pluginName, $this->params, $timer);
Log::debug("PluginsArchiver::%s: %s while archiving %s reports for plugin '%s' %s.",
__FUNCTION__,
$timer->getMemoryLeak(),
$this->params->getPeriod()->getLabel(),
$pluginName,
$this->params->getSegment() ? sprintf("(for segment = '%s')", $this->params->getSegment()->getString()) : ''
);
} catch (Exception $e) {
throw new PluginsArchiverException($e->getMessage() . " - in plugin $pluginName.", $e->getCode(), $e);
} finally {
self::$currentPluginBeingArchived = null;
}
} else {
Log::debug("PluginsArchiver::%s: Not archiving reports for plugin '%s'.", __FUNCTION__, $pluginName);
}
Manager::getInstance()->deleteAll($latestUsedTableId);
unset($archiver);
}
$this->logAggregator->cleanup();
}
public function finalizeArchive()
{
$this->params->logStatusDebug();
$this->archiveWriter->finalizeArchive();
$idArchive = $this->archiveWriter->getIdArchive();
return $idArchive;
}
/**
* Returns if any plugin archiver archives without visits
*/
public static function doesAnyPluginArchiveWithoutVisits()
{
$archivers = static::getPluginArchivers();
foreach ($archivers as $pluginName => $archiverClass) {
if ($archiverClass::shouldRunEvenWhenNoVisits()) {
return true;
}
}
return false;
}
/**
* Loads Archiver class from any plugin that defines one.
*
* @return \Piwik\Plugin\Archiver[]
*/
protected static function getPluginArchivers()
{
if (empty(static::$archivers)) {
$pluginNames = \Piwik\Plugin\Manager::getInstance()->getActivatedPlugins();
$archivers = array();
foreach ($pluginNames as $pluginName) {
$archivers[$pluginName] = self::getPluginArchiverClass($pluginName);
}
static::$archivers = array_filter($archivers);
}
return static::$archivers;
}
private static function getPluginArchiverClass($pluginName)
{
$klassName = 'Piwik\\Plugins\\' . $pluginName . '\\Archiver';
if (class_exists($klassName)
&& is_subclass_of($klassName, 'Piwik\\Plugin\\Archiver')) {
return $klassName;
}
return false;
}
/**
* Whether the specified plugin's reports should be archived
* @param string $pluginName
* @return bool
*/
protected function shouldProcessReportsForPlugin($pluginName)
{
if ($this->params->getRequestedPlugin() == $pluginName) {
return true;
}
if ($this->params->shouldOnlyArchiveRequestedPlugin()) {
return false;
}
if (Rules::shouldProcessReportsAllPlugins(
array($this->params->getSite()->getId()),
$this->params->getSegment(),
$this->params->getPeriod()->getLabel())
) {
return true;
}
if ($this->params->getRequestedPlugin() &&
!\Piwik\Plugin\Manager::getInstance()->isPluginLoaded($this->params->getRequestedPlugin())
) {
return false;
}
return false;
}
protected function aggregateDayVisitsMetrics()
{
$query = $this->archiveProcessor->getLogAggregator()->queryVisitsByDimension();
$data = $query->fetch();
$metrics = $this->convertMetricsIdToName($data);
$this->archiveProcessor->insertNumericRecords($metrics);
return $metrics;
}
protected function convertMetricsIdToName($data)
{
$metrics = array();
foreach ($data as $metricId => $value) {
$readableMetric = Metrics::$mappingFromIdToName[$metricId];
$metrics[$readableMetric] = $value;
}
return $metrics;
}
protected function aggregateMultipleVisitsMetrics()
{
$toSum = Metrics::getVisitsMetricNames();
$metrics = $this->archiveProcessor->aggregateNumericMetrics($toSum);
return $metrics;
}
/**
* @param $archiverClass
* @return Archiver
*/
private function makeNewArchiverObject($archiverClass, $pluginName)
{
$archiver = new $archiverClass($this->archiveProcessor);
/**
* Triggered right after a new **plugin archiver instance** is created.
* Subscribers to this event can configure the plugin archiver, for example prevent the archiving of a plugin's data
* by calling `$archiver->disable()` method.
*
* @param \Piwik\Plugin\Archiver &$archiver The newly created plugin archiver instance.
* @param string $pluginName The name of plugin of which archiver instance was created.
* @param array $this->params Array containing archive parameters (Site, Period, Date and Segment)
* @param bool false This parameter is deprecated and will be removed.
*/
Piwik::postEvent('Archiving.makeNewArchiverObject', array($archiver, $pluginName, $this->params, false));
return $archiver;
}
public static function isArchivingProcessActive()
{
return self::$currentPluginBeingArchived !== null;
}
}

15
matomo/core/ArchiveProcessor/PluginsArchiverException.php Normal file
View file

@ -0,0 +1,15 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\ArchiveProcessor;
class PluginsArchiverException extends \Exception
{
}

384
matomo/core/ArchiveProcessor/Rules.php Normal file
View file

@ -0,0 +1,384 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\ArchiveProcessor;
use Exception;
use Piwik\Config;
use Piwik\Config\GeneralConfig;
use Piwik\DataAccess\ArchiveWriter;
use Piwik\Date;
use Piwik\Log;
use Piwik\Option;
use Piwik\Piwik;
use Piwik\Plugin\Manager;
use Piwik\Plugins\CoreAdminHome\Controller;
use Piwik\Segment;
use Piwik\SettingsPiwik;
use Piwik\SettingsServer;
use Piwik\Site;
use Piwik\Tracker\Cache;
/**
* This class contains Archiving rules/logic which are used when creating and processing Archives.
*
*/
class Rules
{
const OPTION_TODAY_ARCHIVE_TTL = 'todayArchiveTimeToLive';
const OPTION_BROWSER_TRIGGER_ARCHIVING = 'enableBrowserTriggerArchiving';
const FLAG_TABLE_PURGED = 'lastPurge_';
/** Flag that will forcefully disable the archiving process (used in tests only) */
public static $archivingDisabledByTests = false;
/** To disable the Pure Outdated Archive set this to true will skip this task */
public static $disablePureOutdatedArchive = false;
/**
* Returns the name of the archive field used to tell the status of an archive, (ie,
* whether the archive was created successfully or not).
*
* @param array $idSites
* @param Segment $segment
* @param string $periodLabel
* @param string $plugin
* @return string
*/
public static function getDoneStringFlagFor(array $idSites, $segment, $periodLabel, $plugin)
{
if (!empty($plugin)
&& !self::shouldProcessReportsAllPlugins($idSites, $segment, $periodLabel)
) {
return self::getDoneFlagArchiveContainsOnePlugin($segment, $plugin);
}
return self::getDoneFlagArchiveContainsAllPlugins($segment);
}
public static function shouldProcessReportsAllPlugins(array $idSites, Segment $segment, $periodLabel)
{
if (self::isRequestingToAndAbleToForceArchiveSinglePlugin()) {
return false;
}
if ($segment->isEmpty() && ($periodLabel != 'range' || SettingsServer::isArchivePhpTriggered())) {
return true;
}
if ($periodLabel === 'range' && !SettingsServer::isArchivePhpTriggered()) {
return false;
}
return self::isSegmentPreProcessed($idSites, $segment);
}
/**
* @param $idSites
* @return array
*/
public static function getSegmentsToProcess($idSites)
{
$knownSegmentsToArchiveAllSites = SettingsPiwik::getKnownSegmentsToArchive();
$segmentsToProcess = $knownSegmentsToArchiveAllSites;
foreach ($idSites as $idSite) {
$segmentForThisWebsite = SettingsPiwik::getKnownSegmentsToArchiveForSite($idSite);
$segmentsToProcess = array_merge($segmentsToProcess, $segmentForThisWebsite);
}
$segmentsToProcess = array_unique($segmentsToProcess);
return $segmentsToProcess;
}
public static function getDoneFlagArchiveContainsOnePlugin(Segment $segment, $plugin)
{
return 'done' . $segment->getHash() . '.' . $plugin;
}
public static function getDoneFlagArchiveContainsAllPlugins(Segment $segment)
{
return 'done' . $segment->getHash();
}
/**
* Return done flags used to tell how the archiving process for a specific archive was completed,
*
* @param array $plugins
* @param $segment
* @return array
*/
public static function getDoneFlags(array $plugins, Segment $segment)
{
$doneFlags = array();
$doneAllPlugins = self::getDoneFlagArchiveContainsAllPlugins($segment);
$doneFlags[$doneAllPlugins] = $doneAllPlugins;
$plugins = array_unique($plugins);
foreach ($plugins as $plugin) {
$doneOnePlugin = self::getDoneFlagArchiveContainsOnePlugin($segment, $plugin);
$doneFlags[$plugin] = $doneOnePlugin;
}
return $doneFlags;
}
public static function getMinTimeProcessedForInProgressArchive(
Date $dateStart,
\Piwik\Period $period,
Segment $segment,
Site $site
) {
$todayArchiveTimeToLive = self::getPeriodArchiveTimeToLiveDefault($period->getLabel());
$now = time();
$minimumArchiveTime = $now - $todayArchiveTimeToLive;
$idSites = array($site->getId());
$isArchivingDisabled = Rules::isArchivingDisabledFor($idSites, $segment, $period->getLabel());
if ($isArchivingDisabled) {
if ($period->getNumberOfSubperiods() == 0
&& $dateStart->getTimestamp() <= $now
) {
// Today: accept any recent enough archive
$minimumArchiveTime = false;
} else {
// This week, this month, this year:
// accept any archive that was processed today after 00:00:01 this morning
$timezone = $site->getTimezone();
$minimumArchiveTime = Date::factory(Date::factory('now',
$timezone)->getDateStartUTC())->setTimezone($timezone)->getTimestamp();
}
}
return $minimumArchiveTime;
}
public static function setTodayArchiveTimeToLive($timeToLiveSeconds)
{
$timeToLiveSeconds = (int)$timeToLiveSeconds;
if ($timeToLiveSeconds <= 0) {
throw new Exception(Piwik::translate('General_ExceptionInvalidArchiveTimeToLive'));
}
Option::set(self::OPTION_TODAY_ARCHIVE_TTL, $timeToLiveSeconds, $autoLoad = true);
}
public static function getTodayArchiveTimeToLive()
{
$uiSettingIsEnabled = Controller::isGeneralSettingsAdminEnabled();
if ($uiSettingIsEnabled) {
$timeToLive = Option::get(self::OPTION_TODAY_ARCHIVE_TTL);
if ($timeToLive !== false) {
return $timeToLive;
}
}
return self::getTodayArchiveTimeToLiveDefault();
}
public static function getPeriodArchiveTimeToLiveDefault($periodLabel)
{
if (empty($periodLabel) || strtolower($periodLabel) === 'day') {
return self::getTodayArchiveTimeToLive();
}
$config = Config::getInstance();
$general = $config->General;
$key = sprintf('time_before_%s_archive_considered_outdated', $periodLabel);
if (isset($general[$key]) && is_numeric($general[$key]) && $general[$key] > 0) {
return $general[$key];
}
return self::getTodayArchiveTimeToLive();
}
public static function getTodayArchiveTimeToLiveDefault()
{
return Config::getInstance()->General['time_before_today_archive_considered_outdated'];
}
public static function isBrowserArchivingAvailableForSegments()
{
$generalConfig = Config::getInstance()->General;
return !$generalConfig['browser_archiving_disabled_enforce'];
}
public static function isArchivingEnabledFor(array $idSites, Segment $segment, $periodLabel)
{
$isArchivingEnabled = self::isRequestAuthorizedToArchive() && !self::$archivingDisabledByTests;
$generalConfig = Config::getInstance()->General;
if ($periodLabel === 'range') {
if (isset($generalConfig['archiving_range_force_on_browser_request'])
&& $generalConfig['archiving_range_force_on_browser_request'] == false
) {
Log::debug("Not forcing archiving for range period.");
return $isArchivingEnabled;
}
return true;
}
if ($segment->isEmpty()) {
// viewing "All Visits"
return $isArchivingEnabled;
}
if (!$isArchivingEnabled
&& (!self::isBrowserArchivingAvailableForSegments() || self::isSegmentPreProcessed($idSites, $segment))
&& !SettingsServer::isArchivePhpTriggered() // Only applies when we are not running core:archive command
) {
Log::debug("Archiving is disabled because of config setting browser_archiving_disabled_enforce=1 or because the segment is selected to be pre-processed.");
return false;
}
return true;
}
public static function isArchivingDisabledFor(array $idSites, Segment $segment, $periodLabel)
{
return !self::isArchivingEnabledFor($idSites, $segment, $periodLabel);
}
public static function isRequestAuthorizedToArchive(Parameters $params = null)
{
$isRequestAuthorizedToArchive = Rules::isBrowserTriggerEnabled() || SettingsServer::isArchivePhpTriggered();
if (!empty($params)) {
/**
* @ignore
*
* @params bool &$isRequestAuthorizedToArchive
* @params Parameters $params
*/
Piwik::postEvent('Archiving.isRequestAuthorizedToArchive', [&$isRequestAuthorizedToArchive, $params]);
}
return $isRequestAuthorizedToArchive;
}
public static function isBrowserTriggerEnabled()
{
$uiSettingIsEnabled = Controller::isGeneralSettingsAdminEnabled();
if ($uiSettingIsEnabled) {
$browserArchivingEnabled = Option::get(self::OPTION_BROWSER_TRIGGER_ARCHIVING);
if ($browserArchivingEnabled !== false) {
return (bool)$browserArchivingEnabled;
}
}
return (bool)Config::getInstance()->General['enable_browser_archiving_triggering'];
}
public static function setBrowserTriggerArchiving($enabled)
{
if (!is_bool($enabled)) {
throw new Exception('Browser trigger archiving must be set to true or false.');
}
Option::set(self::OPTION_BROWSER_TRIGGER_ARCHIVING, (int)$enabled, $autoLoad = true);
Cache::clearCacheGeneral();
}
/**
* Returns true if the archiving process should skip the calculation of unique visitors
* across several sites. The `[General] enable_processing_unique_visitors_multiple_sites`
* INI config option controls the value of this variable.
*
* @return bool
*/
public static function shouldSkipUniqueVisitorsCalculationForMultipleSites()
{
return Config::getInstance()->General['enable_processing_unique_visitors_multiple_sites'] != 1;
}
/**
* @param array $idSites
* @param Segment $segment
* @return bool
*/
public static function isSegmentPreProcessed(array $idSites, Segment $segment)
{
$segmentsToProcess = self::getSegmentsToProcess($idSites);
if (empty($segmentsToProcess)) {
return false;
}
// If the requested segment is one of the segments to pre-process
// we ensure that any call to the API will trigger archiving of all reports for this segment
$segment = $segment->getString();
// Turns out the getString() above returns the URL decoded segment string
$segmentsToProcessUrlDecoded = array_map('urldecode', $segmentsToProcess);
return in_array($segment, $segmentsToProcess)
|| in_array($segment, $segmentsToProcessUrlDecoded);
}
/**
* Returns done flag values allowed to be selected
*
* @return string[]
*/
public static function getSelectableDoneFlagValues(
$includeInvalidated = true,
Parameters $params = null,
$checkAuthorizedToArchive = true
) {
$possibleValues = array(ArchiveWriter::DONE_OK, ArchiveWriter::DONE_OK_TEMPORARY);
if ($includeInvalidated) {
if (!$checkAuthorizedToArchive || !Rules::isRequestAuthorizedToArchive($params)) {
//If request is not authorized to archive then fetch also invalidated archives
$possibleValues[] = ArchiveWriter::DONE_INVALIDATED;
$possibleValues[] = ArchiveWriter::DONE_PARTIAL;
}
}
return $possibleValues;
}
public static function isRequestingToAndAbleToForceArchiveSinglePlugin()
{
if (!SettingsServer::isArchivePhpTriggered()) {
return false;
}
return !empty($_GET['pluginOnly']) || !empty($_POST['pluginOnly']);
}
public static function isActuallyForceArchivingSinglePlugin()
{
return Loader::getArchivingDepth() <= 1 && self::isRequestingToAndAbleToForceArchiveSinglePlugin();
}
public static function shouldProcessSegmentsWhenReArchivingReports()
{
return Config::getInstance()->General['rearchive_reports_in_past_exclude_segments'] != 1;
}
public static function isSegmentPluginArchivingDisabled($pluginName, $siteId = null)
{
$pluginArchivingSetting = GeneralConfig::getConfigValue('disable_archiving_segment_for_plugins', $siteId);
if (empty($pluginArchivingSetting)) {
return false;
}
if (is_string($pluginArchivingSetting)) {
$pluginArchivingSetting = explode(",", $pluginArchivingSetting);
$pluginArchivingSetting = array_filter($pluginArchivingSetting, function($plugin){
return Manager::getInstance()->isValidPluginName($plugin);
});
}
$pluginArchivingSetting = array_map('strtolower', $pluginArchivingSetting);
return in_array(strtolower($pluginName), $pluginArchivingSetting);
}
}

101
matomo/core/Archiver/Request.php Normal file
View file

@ -0,0 +1,101 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Archiver;
class Request
{
/**
* If a request is aborted, the response of a CliMutli job will be a serialized array containing the
* key/value "aborted => 1".
*/
const ABORT = 'abort';
/**
* @var string
*/
private $url;
/**
* @var callable|null
*/
private $before;
/**
* @param string $url
*/
public function __construct($url)
{
$this->setUrl($url);
}
public function before($callable)
{
$this->before = $callable;
}
public function start()
{
if ($this->before) {
return call_user_func($this->before);
}
}
public function __toString()
{
return $this->url;
}
/**
* @return string
*/
public function getUrl()
{
return $this->url;
}
/**
* @param string $url
*/
public function setUrl($url)
{
$this->url = $url;
}
public function changeDate($newDate)
{
$this->changeParam('date', $newDate);
}
public function makeSureDateIsNotSingleDayRange()
{
// TODO: revisit in matomo 4
// period=range&date=last1/period=range&date=previous1 can cause problems during archiving due to Parameters::isDayArchive()
if (preg_match('/[&?]period=range/', $this->url)) {
if (preg_match('/[&?]date=last1/', $this->url)) {
$this->changeParam('period', 'day');
$this->changeParam('date', 'today');
} else if (preg_match('/[&?]date=previous1/', $this->url)) {
$this->changeParam('period', 'day');
$this->changeParam('date', 'yesterday');
} else if (preg_match('/[&?]date=([^,]+),([^,&]+)/', $this->url, $matches)
&& $matches[1] == $matches[2]
) {
$this->changeParam('period', 'day');
$this->changeParam('date', $matches[1]);
}
}
}
public function changeParam($name, $newValue)
{
$url = $this->getUrl();
$url = preg_replace('/([&?])' . preg_quote($name) . '=[^&]*/', '$1' . $name . '=' . $newValue, $url);
$this->setUrl($url);
}
}

521
matomo/core/AssetManager.php Normal file
View file

@ -0,0 +1,521 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
use Exception;
use Piwik\AssetManager\UIAsset;
use Piwik\AssetManager\UIAsset\InMemoryUIAsset;
use Piwik\AssetManager\UIAsset\OnDiskUIAsset;
use Piwik\AssetManager\UIAssetCacheBuster;
use Piwik\AssetManager\UIAssetFetcher\JScriptUIAssetFetcher;
use Piwik\AssetManager\UIAssetFetcher\StaticUIAssetFetcher;
use Piwik\AssetManager\UIAssetFetcher\StylesheetUIAssetFetcher;
use Piwik\AssetManager\UIAssetFetcher\PluginUmdAssetFetcher;
use Piwik\AssetManager\UIAssetFetcher;
use Piwik\AssetManager\UIAssetMerger\JScriptUIAssetMerger;
use Piwik\AssetManager\UIAssetMerger\StylesheetUIAssetMerger;
use Piwik\Container\StaticContainer;
use Piwik\Plugin\Manager;
/**
* AssetManager is the class used to manage the inclusion of UI assets:
* JavaScript and CSS files.
*
* It performs the following actions:
* - Identifies required assets
* - Includes assets in the rendered HTML page
* - Manages asset merging and minifying
* - Manages server-side cache
*
* Whether assets are included individually or as merged files is defined by
* the global option 'disable_merged_assets'. See the documentation in the global
* config for more information.
*/
class AssetManager extends Singleton
{
const MERGED_CSS_FILE = "asset_manager_global_css.css";
const MERGED_CORE_JS_FILE = "asset_manager_core_js.js";
const MERGED_NON_CORE_JS_FILE = "asset_manager_non_core_js.js";
const CSS_IMPORT_DIRECTIVE = "<link rel=\"stylesheet\" type=\"text/css\" href=\"%s\" />\n";
const JS_IMPORT_DIRECTIVE = "<script type=\"text/javascript\" src=\"%s\"></script>\n";
const JS_DEFER_IMPORT_DIRECTIVE = "<script type=\"text/javascript\" src=\"%s\" defer></script>\n";
const GET_CSS_MODULE_ACTION = "index.php?module=Proxy&action=getCss";
const GET_CORE_JS_MODULE_ACTION = "index.php?module=Proxy&action=getCoreJs";
const GET_NON_CORE_JS_MODULE_ACTION = "index.php?module=Proxy&action=getNonCoreJs";
const GET_JS_UMD_MODULE_ACTION = "index.php?module=Proxy&action=getUmdJs&chunk=";
/**
* @var UIAssetCacheBuster
*/
private $cacheBuster;
/**
* @var UIAssetFetcher
*/
private $minimalStylesheetFetcher;
/**
* @var Theme
*/
private $theme;
public function __construct()
{
$this->cacheBuster = UIAssetCacheBuster::getInstance();
$this->minimalStylesheetFetcher = new StaticUIAssetFetcher(array(), array(), $this->theme);
$theme = Manager::getInstance()->getThemeEnabled();
if (!empty($theme)) {
$this->theme = new Theme();
}
}
/**
* @inheritDoc
* @return AssetManager
*/
public static function getInstance()
{
$assetManager = parent::getInstance();
/**
* Triggered when creating an instance of the asset manager. Lets you overwrite the
* asset manager behavior.
*
* @param AssetManager &$assetManager
*
* @ignore
* This event is not a public event since we don't want to make the asset manager itself public
* API
*/
Piwik::postEvent('AssetManager.makeNewAssetManagerObject', array(&$assetManager));
return $assetManager;
}
/**
* @param UIAssetCacheBuster $cacheBuster
*/
public function setCacheBuster($cacheBuster)
{
$this->cacheBuster = $cacheBuster;
}
/**
* @param UIAssetFetcher $minimalStylesheetFetcher
*/
public function setMinimalStylesheetFetcher($minimalStylesheetFetcher)
{
$this->minimalStylesheetFetcher = $minimalStylesheetFetcher;
}
/**
* @param Theme $theme
*/
public function setTheme($theme)
{
$this->theme = $theme;
}
/**
* Return CSS file inclusion directive(s) using the markup <link>
*
* @return string
*/
public function getCssInclusionDirective()
{
return sprintf(self::CSS_IMPORT_DIRECTIVE, self::GET_CSS_MODULE_ACTION);
}
/**
* Return JS file inclusion directive(s) using the markup <script>
*
* @return string
*/
public function getJsInclusionDirective()
{
$result = "<script type=\"text/javascript\">\n" . StaticContainer::get('Piwik\Translation\Translator')->getJavascriptTranslations() . "\n</script>";
if ($this->isMergedAssetsDisabled()) {
$this->getMergedCoreJSAsset()->delete();
$this->getMergedNonCoreJSAsset()->delete();
$result .= $this->getIndividualCoreAndNonCoreJsIncludes();
} else {
$result .= sprintf(self::JS_IMPORT_DIRECTIVE, self::GET_CORE_JS_MODULE_ACTION);
$result .= sprintf(self::JS_IMPORT_DIRECTIVE, self::GET_NON_CORE_JS_MODULE_ACTION);
$result .= $this->getPluginUmdChunks();
}
return $result;
}
protected function getPluginUmdChunks()
{
$fetcher = $this->getPluginUmdJScriptFetcher();
$chunks = $fetcher->getChunkFiles();
$result = '';
foreach ($chunks as $chunk) {
$src = self::GET_JS_UMD_MODULE_ACTION . urlencode($chunk->getChunkName());
$result .= sprintf(self::JS_DEFER_IMPORT_DIRECTIVE, $src);
}
return $result;
}
/**
* Return the base.less compiled to css
*
* @return UIAsset
*/
public function getCompiledBaseCss()
{
$mergedAsset = new InMemoryUIAsset();
$assetMerger = new StylesheetUIAssetMerger($mergedAsset, $this->minimalStylesheetFetcher, $this->cacheBuster);
$assetMerger->generateFile();
return $mergedAsset;
}
/**
* Return the css merged file absolute location.
* If there is none, the generation process will be triggered.
*
* @return UIAsset
*/
public function getMergedStylesheet()
{
$mergedAsset = $this->getMergedStylesheetAsset();
$assetFetcher = new StylesheetUIAssetFetcher(Manager::getInstance()->getLoadedPluginsName(), $this->theme);
$assetMerger = new StylesheetUIAssetMerger($mergedAsset, $assetFetcher, $this->cacheBuster);
$assetMerger->generateFile();
return $mergedAsset;
}
/**
* Return the core js merged file absolute location.
* If there is none, the generation process will be triggered.
*
* @return UIAsset
*/
public function getMergedCoreJavaScript()
{
return $this->getMergedJavascript($this->getCoreJScriptFetcher(), $this->getMergedCoreJSAsset());
}
/**
* Return the non core js merged file absolute location.
* If there is none, the generation process will be triggered.
*
* @return UIAsset
*/
public function getMergedNonCoreJavaScript()
{
return $this->getMergedJavascript($this->getNonCoreJScriptFetcher(), $this->getMergedNonCoreJSAsset());
}
/**
* Return a chunk JS merged file absolute location.
* If there is none, the generation process will be triggered.
*
* @param string $chunk The name of the chunk. Will either be a plugin name or an integer.
* @return UIAsset
*/
public function getMergedJavaScriptChunk($chunk)
{
$assetFetcher = $this->getPluginUmdJScriptFetcher($chunk);
$outputFile = $assetFetcher->getRequestedChunkOutputFile();
return $this->getMergedJavascript($assetFetcher, $this->getMergedUIAsset($outputFile));
}
/**
* @param boolean|"all" $core
* @return string[]
*/
public function getLoadedPlugins($core)
{
$loadedPlugins = array();
foreach (Manager::getInstance()->getPluginsLoadedAndActivated() as $plugin) {
$pluginName = $plugin->getPluginName();
$pluginIsCore = Manager::getInstance()->isPluginBundledWithCore($pluginName);
if ($core === 'all' || ($pluginIsCore && $core) || (!$pluginIsCore && !$core)) {
$loadedPlugins[] = $pluginName;
}
}
return $loadedPlugins;
}
/**
* Remove previous merged assets
*/
public function removeMergedAssets($pluginName = false)
{
$assetsToRemove = array($this->getMergedStylesheetAsset());
if ($pluginName) {
if ($this->pluginContainsJScriptAssets($pluginName)) {
if (Manager::getInstance()->isPluginBundledWithCore($pluginName)) {
$assetsToRemove[] = $this->getMergedCoreJSAsset();
} else {
$assetsToRemove[] = $this->getMergedNonCoreJSAsset();
}
$assetFetcher = $this->getPluginUmdJScriptFetcher();
foreach ($assetFetcher->getChunkFiles() as $chunk) {
$files = $chunk->getFiles();
$foundInChunk = false;
foreach ($files as $file) {
if (strpos($file, "/$pluginName.umd.") !== false) {
$foundInChunk = true;
}
}
if ($foundInChunk) {
$outputFile = $chunk->getOutputFile();
$asset = $this->getMergedUIAsset($outputFile);
if ($asset->exists()) {
$assetsToRemove[] = $asset;
}
break;
}
}
}
} else {
$assetsToRemove[] = $this->getMergedCoreJSAsset();
$assetsToRemove[] = $this->getMergedNonCoreJSAsset();
$assetFetcher = $this->getPluginUmdJScriptFetcher();
foreach ($assetFetcher->getChunkFiles() as $chunk) {
$outputFile = $chunk->getOutputFile();
$asset = $this->getMergedUIAsset($outputFile);
if ($asset->exists()) {
$assetsToRemove[] = $asset;
}
}
}
$this->removeAssets($assetsToRemove);
}
/**
* Check if the merged file directory exists and is writable.
*
* @return string The directory location
* @throws Exception if directory is not writable.
*/
public function getAssetDirectory()
{
$mergedFileDirectory = StaticContainer::get('path.tmp') . '/assets';
if (!is_dir($mergedFileDirectory)) {
Filesystem::mkdir($mergedFileDirectory);
}
if (!is_writable($mergedFileDirectory)) {
throw new Exception("Directory " . $mergedFileDirectory . " has to be writable.");
}
return $mergedFileDirectory;
}
/**
* Return the global option disable_merged_assets
*
* @return boolean
*/
public function isMergedAssetsDisabled()
{
if (Config::getInstance()->Development['disable_merged_assets'] == 1) {
return true;
}
if (isset($_GET['disable_merged_assets']) && $_GET['disable_merged_assets'] == 1) {
return true;
}
return false;
}
/**
* @param UIAssetFetcher $assetFetcher
* @param UIAsset $mergedAsset
* @return UIAsset
*/
private function getMergedJavascript($assetFetcher, $mergedAsset)
{
$assetMerger = new JScriptUIAssetMerger($mergedAsset, $assetFetcher, $this->cacheBuster);
$assetMerger->generateFile();
return $mergedAsset;
}
/**
* Return individual JS file inclusion directive(s) using the markup <script>
*
* @return string
*/
protected function getIndividualCoreAndNonCoreJsIncludes()
{
return
$this->getIndividualJsIncludesFromAssetFetcher($this->getCoreJScriptFetcher()) .
$this->getIndividualJsIncludesFromAssetFetcher($this->getNonCoreJScriptFetcher()) .
$this->getIndividualJsIncludesFromAssetFetcher($this->getPluginUmdJScriptFetcher());
}
/**
* @param UIAssetFetcher $assetFetcher
* @return string
*/
protected function getIndividualJsIncludesFromAssetFetcher($assetFetcher)
{
$jsIncludeString = '';
$assets = $assetFetcher->getCatalog()->getAssets();
foreach ($assets as $jsFile) {
$jsFile->validateFile();
$jsIncludeString = $jsIncludeString . sprintf(self::JS_IMPORT_DIRECTIVE, $jsFile->getRelativeLocation());
}
return $jsIncludeString;
}
private function getCoreJScriptFetcher()
{
return new JScriptUIAssetFetcher($this->getLoadedPlugins(true), $this->theme);
}
protected function getNonCoreJScriptFetcher()
{
return new JScriptUIAssetFetcher($this->getLoadedPlugins(false), $this->theme);
}
protected function getPluginUmdJScriptFetcher($chunk = null)
{
return new PluginUmdAssetFetcher($this->getLoadedPlugins('all'), $this->theme, $chunk);
}
/**
* @param string $pluginName
* @return boolean
*/
private function pluginContainsJScriptAssets($pluginName)
{
$fetcher = new JScriptUIAssetFetcher(array($pluginName), $this->theme);
try {
$assets = $fetcher->getCatalog()->getAssets();
} catch (\Exception $e) {
// This can happen when a plugin is not valid (eg. Piwik 1.x format)
// When posting the event to the plugin, it returns an exception "Plugin has not been loaded"
return false;
}
$pluginManager = Manager::getInstance();
$plugin = null;
if ($pluginManager->isPluginLoaded($pluginName)) {
$plugin = $pluginManager->getLoadedPlugin($pluginName);
}
if ($plugin && $plugin->isTheme()) {
$theme = $pluginManager->getTheme($pluginName);
$javaScriptFiles = $theme->getJavaScriptFiles();
if (!empty($javaScriptFiles)) {
$assets = array_merge($assets, $javaScriptFiles);
}
}
return !empty($assets);
}
/**
* @param UIAsset[] $uiAssets
*/
public function removeAssets($uiAssets)
{
foreach ($uiAssets as $uiAsset) {
$uiAsset->delete();
}
}
/**
* @return UIAsset
*/
public function getMergedStylesheetAsset()
{
return $this->getMergedUIAsset(self::MERGED_CSS_FILE);
}
/**
* @return UIAsset
*/
private function getMergedCoreJSAsset()
{
return $this->getMergedUIAsset(self::MERGED_CORE_JS_FILE);
}
/**
* @return UIAsset
*/
protected function getMergedNonCoreJSAsset()
{
return $this->getMergedUIAsset(self::MERGED_NON_CORE_JS_FILE);
}
/**
* @param string $fileName
* @return UIAsset
*/
private function getMergedUIAsset($fileName)
{
return new OnDiskUIAsset($this->getAssetDirectory(), $fileName);
}
public static function compileCustomStylesheets($files)
{
$assetManager = new AssetManager();
$fetcher = new StaticUIAssetFetcher($files, $priorityOrder = array(), $theme = null);
$assetManager->setMinimalStylesheetFetcher($fetcher);
return $assetManager->getCompiledBaseCss()->getContent();
}
public static function compileCustomJs($files)
{
$mergedAsset = new InMemoryUIAsset();
$fetcher = new StaticUIAssetFetcher($files, $priorityOrder = array(), $theme = null);
$cacheBuster = UIAssetCacheBuster::getInstance();
$assetMerger = new JScriptUIAssetMerger($mergedAsset, $fetcher, $cacheBuster);
$assetMerger->generateFile();
return $mergedAsset->getContent();
}
}

60
matomo/core/AssetManager/UIAsset.php Normal file
View file

@ -0,0 +1,60 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager;
abstract class UIAsset
{
abstract public function validateFile();
/**
* @return string
*/
abstract public function getAbsoluteLocation();
/**
* @return string
*/
abstract public function getRelativeLocation();
/**
* @return string
*/
abstract public function getBaseDirectory();
/**
* Removes the previous file if it exists.
* Also tries to remove compressed version of the file.
*
* @see ProxyStaticFile::serveStaticFile(serveFile
* @throws Exception if the file couldn't be deleted
*/
abstract public function delete();
/**
* @param string $content
* @throws \Exception
*/
abstract public function writeContent($content);
/**
* @return string
*/
abstract public function getContent();
/**
* @return boolean
*/
abstract public function exists();
/**
* @return int
*/
abstract public function getModificationDate();
}

62
matomo/core/AssetManager/UIAsset/InMemoryUIAsset.php Normal file
View file

@ -0,0 +1,62 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager\UIAsset;
use Exception;
use Piwik\AssetManager\UIAsset;
class InMemoryUIAsset extends UIAsset
{
private $content;
public function validateFile()
{
return;
}
public function getAbsoluteLocation()
{
throw new Exception('invalid operation');
}
public function getRelativeLocation()
{
throw new Exception('invalid operation');
}
public function getBaseDirectory()
{
throw new Exception('invalid operation');
}
public function delete()
{
$this->content = null;
}
public function exists()
{
return false;
}
public function writeContent($content)
{
$this->content = $content;
}
public function getContent()
{
return $this->content;
}
public function getModificationDate()
{
throw new Exception('invalid operation');
}
}

135
matomo/core/AssetManager/UIAsset/OnDiskUIAsset.php Normal file
View file

@ -0,0 +1,135 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager\UIAsset;
use Exception;
use Piwik\AssetManager\UIAsset;
use Piwik\Common;
use Piwik\Filesystem;
class OnDiskUIAsset extends UIAsset
{
/**
* @var string
*/
private $baseDirectory;
/**
* @var string
*/
private $relativeLocation;
/**
* @var string
*/
private $relativeRootDir;
/**
* @param string $baseDirectory
* @param string $fileLocation
*/
public function __construct($baseDirectory, $fileLocation, $relativeRootDir = '')
{
$this->baseDirectory = $baseDirectory;
$this->relativeLocation = $fileLocation;
if (!empty($relativeRootDir)
&& is_string($relativeRootDir)
&& !Common::stringEndsWith($relativeRootDir, '/')) {
$relativeRootDir .= '/';
}
$this->relativeRootDir = $relativeRootDir;
}
public function getAbsoluteLocation()
{
return $this->baseDirectory . '/' . $this->relativeLocation;
}
public function getRelativeLocation()
{
if (isset($this->relativeRootDir)) {
return $this->relativeRootDir . $this->relativeLocation;
}
return $this->relativeLocation;
}
public function getBaseDirectory()
{
return $this->baseDirectory;
}
public function validateFile()
{
if (!$this->assetIsReadable()) {
throw new Exception("The ui asset with 'href' = " . $this->getAbsoluteLocation() . " is not readable");
}
}
public function delete()
{
if ($this->exists()) {
try {
Filesystem::remove($this->getAbsoluteLocation());
} catch (Exception $e) {
throw new Exception("Unable to delete merged file : " . $this->getAbsoluteLocation() . ". Please delete the file and refresh");
}
// try to remove compressed version of the merged file.
Filesystem::remove($this->getAbsoluteLocation() . ".deflate", true);
Filesystem::remove($this->getAbsoluteLocation() . ".gz", true);
}
}
/**
* @param string $content
* @throws \Exception
*/
public function writeContent($content)
{
$this->delete();
$newFile = @fopen($this->getAbsoluteLocation(), "w");
if (!$newFile) {
throw new Exception("The file : " . $newFile . " can not be opened in write mode.");
}
fwrite($newFile, $content);
fclose($newFile);
}
/**
* @return string
*/
public function getContent()
{
return file_get_contents($this->getAbsoluteLocation());
}
public function exists()
{
return $this->assetIsReadable();
}
/**
* @return boolean
*/
private function assetIsReadable()
{
return is_readable($this->getAbsoluteLocation());
}
public function getModificationDate()
{
return filemtime($this->getAbsoluteLocation());
}
}

69
matomo/core/AssetManager/UIAssetCacheBuster.php Normal file
View file

@ -0,0 +1,69 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
* @method static \Piwik\AssetManager\UIAssetCacheBuster getInstance()
*/
namespace Piwik\AssetManager;
use Piwik\Plugin\Manager;
use Piwik\Singleton;
use Piwik\Version;
class UIAssetCacheBuster extends Singleton
{
/**
* Cache buster based on
* - Piwik version
* - Loaded plugins (name and version)
* - Super user salt
* - Latest
*
* @param string[] $pluginNames
* @return string
*/
public function piwikVersionBasedCacheBuster($pluginNames = false)
{
static $cachedCacheBuster = null;
if (empty($cachedCacheBuster) || $pluginNames !== false) {
$masterFile = PIWIK_INCLUDE_PATH . '/.git/refs/heads/master';
$currentGitHash = file_exists($masterFile) ? @file_get_contents($masterFile) : '';
$manager = Manager::getInstance();
$plugins = !$pluginNames ? $manager->getActivatedPlugins() : $pluginNames;
sort($plugins);
$pluginsInfo = '';
foreach ($plugins as $pluginName) {
if ($manager->isPluginLoaded($pluginName)) {
$plugin = $manager->getLoadedPlugin($pluginName);
$pluginsInfo .= $plugin->getPluginName() . $plugin->getVersion() . ',';
}
}
$cacheBuster = md5($pluginsInfo . PHP_VERSION . Version::VERSION . trim($currentGitHash ?? ''));
if ($pluginNames !== false) {
return $cacheBuster;
}
$cachedCacheBuster = $cacheBuster;
}
return $cachedCacheBuster;
}
/**
* @param string $content
* @return string
*/
public function md5BasedCacheBuster($content)
{
return md5($content);
}
}

73
matomo/core/AssetManager/UIAssetCatalog.php Normal file
View file

@ -0,0 +1,73 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager;
class UIAssetCatalog
{
/**
* @var UIAsset[]
*/
private $uiAssets = array();
/**
* @var UIAssetCatalogSorter
*/
private $catalogSorter;
/**
* @var string[] Absolute file locations
*/
private $existingAssetLocations = array();
/**
* @param UIAssetCatalogSorter $catalogSorter
*/
public function __construct($catalogSorter)
{
$this->catalogSorter = $catalogSorter;
}
/**
* @param UIAsset $uiAsset
*/
public function addUIAsset($uiAsset)
{
$location = $uiAsset->getAbsoluteLocation();
if (!$this->assetAlreadyInCatalog($location)) {
$this->existingAssetLocations[] = $location;
$this->uiAssets[] = $uiAsset;
}
}
/**
* @return UIAsset[]
*/
public function getAssets()
{
return $this->uiAssets;
}
/**
* @return UIAssetCatalog
*/
public function getSortedCatalog()
{
return $this->catalogSorter->sortUIAssetCatalog($this);
}
/**
* @param UIAsset $uiAsset
* @return boolean
*/
private function assetAlreadyInCatalog($location)
{
return in_array($location, $this->existingAssetLocations);
}
}

58
matomo/core/AssetManager/UIAssetCatalogSorter.php Normal file
View file

@ -0,0 +1,58 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager;
class UIAssetCatalogSorter
{
/**
* @var string[]
*/
private $priorityOrder;
/**
* @param string[] $priorityOrder
*/
public function __construct($priorityOrder)
{
$this->priorityOrder = $priorityOrder;
}
/**
* @param UIAssetCatalog $uiAssetCatalog
* @return UIAssetCatalog
*/
public function sortUIAssetCatalog($uiAssetCatalog)
{
$sortedCatalog = new UIAssetCatalog($this);
foreach ($this->priorityOrder as $filePattern) {
$assetsMatchingPattern = array_filter($uiAssetCatalog->getAssets(), function ($uiAsset) use ($filePattern) {
return preg_match('~^' . $filePattern . '~', $uiAsset->getRelativeLocation());
});
foreach ($assetsMatchingPattern as $assetMatchingPattern) {
$sortedCatalog->addUIAsset($assetMatchingPattern);
}
}
$this->addUnmatchedAssets($uiAssetCatalog, $sortedCatalog);
return $sortedCatalog;
}
/**
* @param UIAssetCatalog $uiAssetCatalog
* @param UIAssetCatalog $sortedCatalog
*/
private function addUnmatchedAssets($uiAssetCatalog, $sortedCatalog)
{
foreach ($uiAssetCatalog->getAssets() as $uiAsset) {
$sortedCatalog->addUIAsset($uiAsset);
}
}
}

196
matomo/core/AssetManager/UIAssetFetcher.php Normal file
View file

@ -0,0 +1,196 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager;
use Piwik\AssetManager\UIAsset\OnDiskUIAsset;
use Piwik\Plugin\Manager;
use Piwik\Theme;
abstract class UIAssetFetcher
{
/**
* @var UIAssetCatalog
*/
protected $catalog;
/**
* @var string[]
*/
protected $fileLocations = array();
/**
* @var string[]
*/
protected $plugins;
/**
* @var Theme
*/
private $theme;
/**
* @param string[] $plugins
* @param Theme $theme
*/
public function __construct($plugins, $theme)
{
$this->plugins = $plugins;
$this->theme = $theme;
}
/**
* @return string[]
*/
public function getPlugins()
{
return $this->plugins;
}
/**
* $return UIAssetCatalog
*/
public function getCatalog()
{
if ($this->catalog == null) {
$this->createCatalog();
}
return $this->catalog;
}
abstract protected function retrieveFileLocations();
/**
* @return string[]
*/
abstract protected function getPriorityOrder();
private function createCatalog()
{
$this->retrieveFileLocations();
$this->initCatalog();
$this->populateCatalog();
$this->sortCatalog();
}
private function initCatalog()
{
$catalogSorter = new UIAssetCatalogSorter($this->getPriorityOrder());
$this->catalog = new UIAssetCatalog($catalogSorter);
}
private function populateCatalog()
{
$pluginBaseDir = Manager::getPluginsDirectory();
$pluginWebDirectories = Manager::getAlternativeWebRootDirectories();
$matomoRootDir = $this->getBaseDirectory();
foreach ($this->fileLocations as $fileLocation) {
$fileAbsolute = $matomoRootDir . '/' . $fileLocation;
$newUIAsset = new OnDiskUIAsset($this->getBaseDirectory(), $fileLocation);
if ($newUIAsset->exists()) {
$this->catalog->addUIAsset($newUIAsset);
continue;
}
$found = false;
if (strpos($fileAbsolute, $pluginBaseDir) === 0) {
// we iterate over all custom plugin directories only for plugin files, not libs files (not needed there)
foreach ($pluginWebDirectories as $pluginDirectory => $relative) {
$fileTest = str_replace($pluginBaseDir, $pluginDirectory, $fileAbsolute);
$newFileRelative = str_replace($pluginDirectory, '', $fileTest);
$testAsset = new OnDiskUIAsset($pluginDirectory, $newFileRelative, $relative);
if ($testAsset->exists()) {
$this->catalog->addUIAsset($testAsset);
$found = true;
break;
}
}
}
if (!$found) {
// we add it anyway so it'll trigger an error about the missing file
$this->catalog->addUIAsset($newUIAsset);
}
}
}
private function sortCatalog()
{
$this->catalog = $this->catalog->getSortedCatalog();
}
/**
* @return string
*/
private function getBaseDirectory()
{
// served by web server directly, so must be a public path
return PIWIK_DOCUMENT_ROOT;
}
/**
* @return Theme
*/
public function getTheme()
{
return $this->theme;
}
public static $bowerComponentFileMappings = [
'libs/bower_components/jquery/dist/jquery.min.js' => 'node_modules/jquery/dist/jquery.min.js',
'libs/bower_components/jquery-ui/ui/minified/jquery-ui.min.js' => 'node_modules/jquery-ui-dist/jquery-ui.min.js',
"libs/bower_components/sprintf/dist/sprintf.min.js" => "node_modules/sprintf-js/dist/sprintf.min.js",
"libs/bower_components/materialize/dist/js/materialize.min.js" => "node_modules/materialize-css/dist/js/materialize.min.js",
"libs/bower_components/jquery.scrollTo/jquery.scrollTo.min.js" => "node_modules/jquery.scrollto/jquery.scrollTo.min.js",
"libs/bower_components/mousetrap/mousetrap.min.js" => "node_modules/mousetrap/mousetrap.min.js",
"libs/bower_components/angular/angular.min.js" => 'node_modules/angular/angular.min.js',
"libs/bower_components/angular-sanitize/angular-sanitize.min.js" => "node_modules/angular-sanitize/angular-sanitize.min.js",
"libs/bower_components/angular-animate/angular-animate.min.js" => "node_modules/angular-animate/angular-animate.min.js",
"libs/bower_components/angular-cookies/angular-cookies.min.js" => "node_modules/angular-cookies/angular-cookies.min.js",
"libs/bower_components/ngDialog/js/ngDialog.min.js" => "node_modules/ng-dialog/js/ngDialog.min.js",
"libs/bower_components/jQuery.dotdotdot/src/js/jquery.dotdotdot.min.js" => "node_modules/jquery.dotdotdot/dist/jquery.dotdotdot.js",
"libs/bower_components/visibilityjs/lib/visibility.core.js" => "node_modules/visibilityjs/lib/visibility.core.js",
"libs/bower_components/iframe-resizer/js/iframeResizer.min.js" => "node_modules/iframe-resizer/js/iframeResizer.min.js",
"libs/bower_components/qrcode.js/qrcode.js" => "node_modules/qrcodejs2/qrcode.min.js",
"libs/bower_components/chroma-js/chroma.min.js" => "node_modules/chroma-js/chroma.min.js",
"libs/jquery/jquery.browser.js" => "node_modules/jquery.browser/dist/jquery.browser.min.js",
"plugins/CoreHome/angularjs/dialogtoggler/dialogtoggler.directive.js" => null,
"plugins/CoreHome/angularjs/dialogtoggler/dialogtoggler.controller.js" => null,
"plugins/CoreHome/angularjs/dialogtoggler/dialogtoggler-urllistener.service.js" => null,
"libs/jquery/jquery.truncate.js" => null,
"libs/jquery/themes/base/jquery-ui.min.css" => "node_modules/jquery-ui-dist/jquery-ui.min.css",
"libs/bower_components/materialize/dist/css/materialize.min.css" => "node_modules/materialize-css/dist/css/materialize.min.css",
"node_modules/jquery-ui-dist/jquery-ui.theme.min.css" => "node_modules/jquery-ui-dist/jquery-ui.theme.min.css",
"libs/bower_components/ngDialog/css/ngDialog.min.css" => null,
"libs/bower_components/ngDialog/css/ngDialog-theme-default.min.css" => null,
"plugins/CoreHome/angularjs/dialogtoggler/ngdialog.less" => null,
];
protected function mapBowerComponentFilesForBC(array &$fileLocations)
{
foreach ($fileLocations as $index => $location) {
if (!isset(self::$bowerComponentFileMappings[$location])) {
continue;
}
if (self::$bowerComponentFileMappings[$location] === null) {
unset($fileLocations[$index]);
} else {
$fileLocations[$index] = self::$bowerComponentFileMappings[$location];
}
}
}
}

69
matomo/core/AssetManager/UIAssetFetcher/Chunk.php Normal file
View file

@ -0,0 +1,69 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager\UIAssetFetcher;
class Chunk
{
/**
* @var string
*/
private $chunkName;
/**
* @var string[]
*/
private $files;
public function __construct($chunkName, $files)
{
$this->chunkName = $chunkName;
$this->files = $files;
}
/**
* @return string
*/
public function getOutputFile(): string
{
return "asset_manager_chunk.{$this->chunkName}.js";
}
/**
* @return string[]
*/
public function getFiles(): array
{
return $this->files;
}
/**
* @param string[] $files
*/
public function setFiles(array $files): void
{
$this->files = $files;
}
/**
* @return string
*/
public function getChunkName(): string
{
return $this->chunkName;
}
/**
* @param string $chunkName
*/
public function setChunkName(string $chunkName): void
{
$this->chunkName = $chunkName;
}
}

95
matomo/core/AssetManager/UIAssetFetcher/JScriptUIAssetFetcher.php Normal file
View file

@ -0,0 +1,95 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager\UIAssetFetcher;
use Piwik\AssetManager\UIAssetFetcher;
use Piwik\Piwik;
class JScriptUIAssetFetcher extends UIAssetFetcher
{
protected function retrieveFileLocations()
{
if (!empty($this->plugins)) {
/**
* Triggered when gathering the list of all JavaScript files needed by Piwik
* and its plugins.
*
* Plugins that have their own JavaScript should use this event to make those
* files load in the browser.
*
* JavaScript files should be placed within a **javascripts** subdirectory in your
* plugin's root directory.
*
* _Note: While you are developing your plugin you should enable the config setting
* `[Development] disable_merged_assets` so JavaScript files will be reloaded immediately
* after every change._
*
* **Example**
*
* public function getJsFiles(&$jsFiles)
* {
* $jsFiles[] = "plugins/MyPlugin/javascripts/myfile.js";
* $jsFiles[] = "plugins/MyPlugin/javascripts/anotherone.js";
* }
*
* @param string[] $jsFiles The JavaScript files to load.
*/
Piwik::postEvent('AssetManager.getJavaScriptFiles', array(&$this->fileLocations), null, $this->plugins);
}
$this->addThemeFiles();
$this->mapBowerComponentFilesForBC($this->fileLocations);
}
protected function addThemeFiles()
{
$theme = $this->getTheme();
if (!$theme) {
return;
}
if (in_array($theme->getThemeName(), $this->plugins)) {
$jsInThemes = $this->getTheme()->getJavaScriptFiles();
if (!empty($jsInThemes)) {
foreach ($jsInThemes as $jsFile) {
$this->fileLocations[] = $jsFile;
}
}
}
}
protected function getPriorityOrder()
{
return array(
'node_modules/jquery/dist/jquery.min.js',
'node_modules/jquery/dist/jquery.js',
'node_modules/materialize-css/dist/js/materialize.min.js', // so jquery ui datepicker overrides materializecss
'node_modules/jquery-ui-dist/jquery-ui.min.js',
'node_modules/jquery-ui-dist/jquery-ui.js',
"plugins/CoreHome/javascripts/materialize-bc.js",
"node_modules/jquery.browser/dist/jquery.browser.min.js",
'node_modules/',
'libs/',
'js/',
'plugins/CoreVue/polyfills/dist/MatomoPolyfills',
'piwik.js',
'matomo.js',
'plugins/CoreHome/javascripts/require.js',
'plugins/Morpheus/javascripts/piwikHelper.js',
'plugins/Morpheus/javascripts/',
'plugins/CoreHome/javascripts/uiControl.js',
'plugins/CoreHome/javascripts/broadcast.js',
'plugins/CoreHome/javascripts/', // load CoreHome JS before other plugins
'plugins/',
'tests/',
);
}
}

312
matomo/core/AssetManager/UIAssetFetcher/PluginUmdAssetFetcher.php Normal file
View file

@ -0,0 +1,312 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager\UIAssetFetcher;
use Piwik\AssetManager\UIAssetFetcher;
use Piwik\Cache;
use Piwik\Config;
use Piwik\Development;
use Piwik\Plugin\Manager;
class PluginUmdAssetFetcher extends UIAssetFetcher
{
/**
* @var string
*/
private $requestedChunk;
/**
* @var boolean
*/
private $loadIndividually;
/**
* @var int|null
*/
private $chunkCount;
public function __construct($plugins, $theme, $chunk, $loadIndividually = null, $chunkCount = null)
{
parent::__construct($plugins, $theme);
if ($loadIndividually === null) {
$loadIndividually = self::getDefaultLoadIndividually();
}
if ($chunkCount === null) {
$chunkCount = self::getDefaultChunkCount();
}
$this->requestedChunk = $chunk;
$this->loadIndividually = $loadIndividually;
$this->chunkCount = $chunkCount;
if (!$this->loadIndividually && (!is_int($chunkCount) || $chunkCount <= 0)) {
throw new \Exception("Invalid chunk count: $chunkCount");
}
}
public function getRequestedChunkOutputFile()
{
return "asset_manager_chunk.{$this->requestedChunk}.js";
}
/**
* @return Chunk[]
*/
public function getChunkFiles()
{
$allPluginUmds = $this->getAllPluginUmds();
if ($this->loadIndividually) {
return $allPluginUmds;
}
$totalSize = $this->getTotalChunkSize($allPluginUmds);
$chunkFiles = $this->dividePluginUmdsByChunkCount($allPluginUmds, $totalSize);
$chunks = [];
foreach ($chunkFiles as $index => $jsFiles) {
$chunks[] = new Chunk($index, $jsFiles);
}
return $chunks;
}
private function getTotalChunkSize($allPluginUmds)
{
$totalSize = 0;
foreach ($allPluginUmds as $chunk) {
$path = PIWIK_INCLUDE_PATH . '/' . $chunk->getFiles()[0];
if (is_file($path)) {
$totalSize += filesize($path);
}
}
return $totalSize;
}
private function getAllPluginUmds()
{
$plugins = self::orderPluginsByPluginDependencies($this->plugins, false);
$allPluginUmds = [];
foreach ($plugins as $plugin) {
$pluginDir = self::getRelativePluginDirectory($plugin);
$minifiedUmd = "$pluginDir/vue/dist/$plugin.umd.min.js";
if (!is_file(PIWIK_INCLUDE_PATH . '/' . $minifiedUmd)) {
continue;
}
$allPluginUmds[] = new Chunk($plugin, [$minifiedUmd]);
}
return $allPluginUmds;
}
private function dividePluginUmdsByChunkCount($allPluginUmds, $totalSize)
{
$chunkSizeLimit = floor($totalSize / $this->chunkCount);
$chunkFiles = [];
$currentChunkIndex = 0;
$currentChunkSize = 0;
foreach ($allPluginUmds as $pluginChunk) {
$path = PIWIK_INCLUDE_PATH . '/' . $pluginChunk->getFiles()[0];
if (!is_file($path)) {
continue;
}
$size = filesize($path);
$currentChunkSize += $size;
if ($currentChunkSize > $chunkSizeLimit
&& !empty($chunkFiles[$currentChunkIndex])
&& $currentChunkIndex < $this->chunkCount - 1
) {
++$currentChunkIndex;
$currentChunkSize = $size;
}
$chunkFiles[$currentChunkIndex][] = $pluginChunk->getFiles()[0];
}
return $chunkFiles;
}
protected function retrieveFileLocations()
{
if (empty($this->plugins)) {
return;
}
if ($this->requestedChunk !== null && $this->requestedChunk !== '') {
$chunkFiles = $this->getChunkFiles();
$foundChunk = null;
foreach ($chunkFiles as $chunk) {
if ($chunk->getChunkName() == $this->requestedChunk) {
$foundChunk = $chunk;
break;
}
}
if (!$foundChunk) {
throw new \Exception("Could not find chunk {$this->requestedChunk}");
}
foreach ($foundChunk->getFiles() as $file) {
$this->fileLocations[] = $file;
}
return;
}
// either loadFilesIndividually = true, or being called w/ disable_merged_assets=1
$this->addUmdFilesIfDetected($this->plugins);
}
private function addUmdFilesIfDetected($plugins)
{
$plugins = self::orderPluginsByPluginDependencies($plugins, false);
foreach ($plugins as $plugin) {
$fileLocation = self::getUmdFileToUseForPlugin($plugin);
if ($fileLocation) {
$this->fileLocations[] = $fileLocation;
}
}
}
public static function getUmdFileToUseForPlugin($plugin)
{
$pluginDir = self::getRelativePluginDirectory($plugin);
$devUmd = "$pluginDir/vue/dist/$plugin.development.umd.js";
$minifiedUmd = "$pluginDir/vue/dist/$plugin.umd.min.js";
$umdSrcFolder = "$pluginDir/vue/src";
// in case there are dist files but no src files, which can happen during development
if (is_dir(PIWIK_INCLUDE_PATH . '/' . $umdSrcFolder)) {
if (Development::isEnabled() && is_file(PIWIK_INCLUDE_PATH . '/' . $devUmd)) {
return $devUmd;
} else if (is_file(PIWIK_INCLUDE_PATH . '/' . $minifiedUmd)) {
return $minifiedUmd;
}
}
return null;
}
public static function orderPluginsByPluginDependencies($plugins, $keepUnresolved = true)
{
$result = [];
while (!empty($plugins)) {
self::visitPlugin(reset($plugins), $keepUnresolved, $plugins, $result);
}
return $result;
}
public static function getPluginDependencies($plugin)
{
$pluginDir = self::getPluginDirectory($plugin);
$umdMetadata = "$pluginDir/vue/dist/umd.metadata.json";
$cache = Cache::getTransientCache();
$cacheKey = 'PluginUmdAssetFetcher.pluginDependencies.' . $plugin;
$pluginDependencies = $cache->fetch($cacheKey);
if (!is_array($pluginDependencies)) {
$pluginDependencies = [];
if (is_file($umdMetadata)) {
$pluginDependencies = json_decode(file_get_contents($umdMetadata), true);
$pluginDependencies = $pluginDependencies['dependsOn'] ?? [];
}
$cache->save($cacheKey, $pluginDependencies);
}
return $cache->fetch($cacheKey);
}
private static function visitPlugin($plugin, $keepUnresolved, &$plugins, &$result)
{
// remove the plugin from the array of plugins to visit
$index = array_search($plugin, $plugins);
if ($index !== false) {
unset($plugins[$index]);
} else {
return; // already visited
}
// read the plugin dependencies, if any
$pluginDependencies = self::getPluginDependencies($plugin);
if (!empty($pluginDependencies)) {
// visit each plugin this one depends on first, so it is loaded first
foreach ($pluginDependencies as $pluginDependency) {
// check if dependency is not activated
if (!in_array($pluginDependency, $plugins)
&& !in_array($pluginDependency, $result)
&& !$keepUnresolved
) {
return;
}
self::visitPlugin($pluginDependency, $keepUnresolved, $plugins, $result);
}
}
// add the plugin to the load order after visiting its dependencies
$result[] = $plugin;
}
protected function getPriorityOrder()
{
// the JS files are already ordered properly so this result doesn't matter
return [];
}
private static function getRelativePluginDirectory($plugin)
{
$result = self::getPluginDirectory($plugin);
$matomoPath = rtrim(PIWIK_INCLUDE_PATH, '/') . '/';
$webroots = array_merge(
Manager::getAlternativeWebRootDirectories(),
[$matomoPath => '/']
);
foreach ($webroots as $webrootAbsolute => $webrootRelative) {
if (strpos($result, $webrootAbsolute) === 0) {
$result = str_replace($webrootAbsolute, $webrootRelative, $result);
break;
}
}
$result = ltrim($result, '/');
return $result;
}
private static function getPluginDirectory($plugin)
{
return Manager::getInstance()->getPluginDirectory($plugin);
}
public static function getDefaultLoadIndividually()
{
return (Config::getInstance()->General['assets_umd_load_individually'] ?? 0) == 1;
}
public static function getDefaultChunkCount()
{
return (int)(Config::getInstance()->General['assets_umd_chunk_count'] ?? 3);
}
}

36
matomo/core/AssetManager/UIAssetFetcher/StaticUIAssetFetcher.php Normal file
View file

@ -0,0 +1,36 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager\UIAssetFetcher;
use Piwik\AssetManager\UIAssetFetcher;
class StaticUIAssetFetcher extends UIAssetFetcher
{
/**
* @var string[]
*/
private $priorityOrder;
public function __construct($fileLocations, $priorityOrder, $theme)
{
parent::__construct(array(), $theme);
$this->fileLocations = $fileLocations;
$this->priorityOrder = $priorityOrder;
}
protected function retrieveFileLocations()
{
}
protected function getPriorityOrder()
{
return $this->priorityOrder;
}
}

101
matomo/core/AssetManager/UIAssetFetcher/StylesheetUIAssetFetcher.php Normal file
View file

@ -0,0 +1,101 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager\UIAssetFetcher;
use Piwik\AssetManager\UIAssetFetcher;
use Piwik\Piwik;
class StylesheetUIAssetFetcher extends UIAssetFetcher
{
protected function getPriorityOrder()
{
$theme = $this->getTheme();
$themeName = $theme->getThemeName();
$order = array(
'plugins/Morpheus/stylesheets/base/bootstrap.css',
'plugins/Morpheus/stylesheets/base/icons.css',
'node_modules/',
'libs/',
'plugins/CoreHome/stylesheets/color_manager.css', // must be before other Piwik stylesheets
'plugins/Morpheus/stylesheets/base.less',
);
if ($themeName === 'Morpheus') {
$order[] = 'plugins\/((?!Morpheus).)*\/';
} else {
$order[] = sprintf('plugins\/((?!(Morpheus)|(%s)).)*\/', $themeName);
}
$order = array_merge(
$order,
array(
'plugins/Dashboard/stylesheets/dashboard.less',
'tests/',
)
);
return $order;
}
protected function retrieveFileLocations()
{
/**
* Triggered when gathering the list of all stylesheets (CSS and LESS) needed by
* Piwik and its plugins.
*
* Plugins that have stylesheets should use this event to make those stylesheets
* load.
*
* Stylesheets should be placed within a **stylesheets** subdirectory in your plugin's
* root directory.
*
* **Example**
*
* public function getStylesheetFiles(&$stylesheets)
* {
* $stylesheets[] = "plugins/MyPlugin/stylesheets/myfile.less";
* $stylesheets[] = "plugins/MyPlugin/stylesheets/myotherfile.css";
* }
*
* @param string[] &$stylesheets The list of stylesheet paths.
*/
Piwik::postEvent('AssetManager.getStylesheetFiles', array(&$this->fileLocations));
$this->addUmdCssFilesIfDetected($this->plugins);
$this->addThemeFiles();
$this->mapBowerComponentFilesForBC($this->fileLocations);
}
protected function addThemeFiles()
{
$theme = $this->getTheme();
if (!$theme) {
return;
}
$themeStylesheet = $theme->getStylesheet();
if ($themeStylesheet) {
$this->fileLocations[] = $themeStylesheet;
}
}
private function addUmdCssFilesIfDetected(array $plugins)
{
foreach ($plugins as $plugin) {
$css = "plugins/$plugin/vue/dist/$plugin.css";
if (is_file(PIWIK_INCLUDE_PATH . '/' . $css)) {
$this->fileLocations[] = $css;
}
}
}
}

198
matomo/core/AssetManager/UIAssetMerger.php Normal file
View file

@ -0,0 +1,198 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager;
abstract class UIAssetMerger
{
/**
* @var UIAssetFetcher
*/
private $assetFetcher;
/**
* @var UIAsset
*/
private $mergedAsset;
/**
* @var string
*/
protected $mergedContent;
/**
* @var UIAssetCacheBuster
*/
protected $cacheBuster;
/**
* @var string
*/
protected $cacheBusterValue;
/**
* @param UIAsset $mergedAsset
* @param UIAssetFetcher $assetFetcher
* @param UIAssetCacheBuster $cacheBuster
*/
public function __construct($mergedAsset, $assetFetcher, $cacheBuster)
{
$this->mergedAsset = $mergedAsset;
$this->assetFetcher = $assetFetcher;
$this->cacheBuster = $cacheBuster;
}
public function generateFile()
{
if (!$this->shouldGenerate()) {
return;
}
$this->mergedContent = $this->getMergedAssets();
$this->postEvent($this->mergedContent);
$this->adjustPaths();
$this->addPreamble();
$this->writeContentToFile();
}
/**
* @return string
*/
abstract protected function getMergedAssets();
/**
* @return string
*/
abstract protected function generateCacheBuster();
/**
* @return string
*/
abstract protected function getPreamble();
/**
* @return string
*/
abstract protected function getFileSeparator();
/**
* @param UIAsset $uiAsset
* @return string
*/
abstract protected function processFileContent($uiAsset);
/**
* @param string $mergedContent
*/
abstract protected function postEvent(&$mergedContent);
protected function getConcatenatedAssets()
{
if (empty($this->mergedContent)) {
$this->concatenateAssets();
}
return $this->mergedContent;
}
protected function concatenateAssets()
{
$mergedContent = '';
foreach ($this->getAssetCatalog()->getAssets() as $uiAsset) {
$uiAsset->validateFile();
$content = $this->processFileContent($uiAsset);
$mergedContent .= $this->getFileSeparator() . $content;
}
$this->mergedContent = $mergedContent;
}
/**
* @return string[]
*/
protected function getPlugins()
{
return $this->assetFetcher->getPlugins();
}
/**
* @return UIAssetCatalog
*/
protected function getAssetCatalog()
{
return $this->assetFetcher->getCatalog();
}
/**
* @return boolean
*/
private function shouldGenerate()
{
if (!$this->mergedAsset->exists()) {
return true;
}
return !$this->isFileUpToDate();
}
/**
* @return boolean
*/
private function isFileUpToDate()
{
$f = fopen($this->mergedAsset->getAbsoluteLocation(), 'r');
$firstLine = fgets($f);
fclose($f);
if (!empty($firstLine) && trim($firstLine) == trim($this->getCacheBusterValue())) {
return true;
}
// Some CSS file in the merge, has changed since last merged asset was generated
// Note: we do not detect changes in @import'ed LESS files
return false;
}
private function adjustPaths()
{
$theme = $this->assetFetcher->getTheme();
// During installation theme is not yet ready
if ($theme) {
$this->mergedContent = $this->assetFetcher->getTheme()->rewriteAssetsPathToTheme($this->mergedContent);
}
}
private function writeContentToFile()
{
$this->mergedAsset->writeContent($this->mergedContent);
}
/**
* @return string
*/
protected function getCacheBusterValue()
{
if (empty($this->cacheBusterValue)) {
$this->cacheBusterValue = $this->generateCacheBuster();
}
return $this->cacheBusterValue;
}
private function addPreamble()
{
$this->mergedContent = $this->getPreamble() . $this->mergedContent;
}
}

87
matomo/core/AssetManager/UIAssetMerger/JScriptUIAssetMerger.php Normal file
View file

@ -0,0 +1,87 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager\UIAssetMerger;
use Piwik\AssetManager\UIAsset;
use Piwik\AssetManager\UIAssetCacheBuster;
use Piwik\AssetManager\UIAssetFetcher\JScriptUIAssetFetcher;
use Piwik\AssetManager\UIAssetMerger;
use Piwik\AssetManager\UIAssetMinifier;
use Piwik\Piwik;
class JScriptUIAssetMerger extends UIAssetMerger
{
/**
* @var UIAssetMinifier
*/
private $assetMinifier;
/**
* @param UIAsset $mergedAsset
* @param JScriptUIAssetFetcher $assetFetcher
* @param UIAssetCacheBuster $cacheBuster
*/
public function __construct($mergedAsset, $assetFetcher, $cacheBuster)
{
parent::__construct($mergedAsset, $assetFetcher, $cacheBuster);
$this->assetMinifier = UIAssetMinifier::getInstance();
}
protected function getMergedAssets()
{
return $this->getConcatenatedAssets();
}
protected function generateCacheBuster()
{
$cacheBuster = $this->cacheBuster->piwikVersionBasedCacheBuster($this->getPlugins());
return "/* Matomo Javascript - cb=" . $cacheBuster . "*/\n";
}
protected function getPreamble()
{
return $this->getCacheBusterValue();
}
protected function postEvent(&$mergedContent)
{
$plugins = $this->getPlugins();
if (!empty($plugins)) {
/**
* Triggered after all the JavaScript files Piwik uses are minified and merged into a
* single file, but before the merged JavaScript is written to disk.
*
* Plugins can use this event to modify merged JavaScript or do something else
* with it.
*
* @param string $mergedContent The minified and merged JavaScript.
*/
Piwik::postEvent('AssetManager.filterMergedJavaScripts', array(&$mergedContent), null, $plugins);
}
}
public function getFileSeparator()
{
return "\n";
}
protected function processFileContent($uiAsset)
{
$content = $uiAsset->getContent();
if (!$this->assetMinifier->isMinifiedJs($content)) {
$content = $this->assetMinifier->minifyJs($content);
}
return $content;
}
}

272
matomo/core/AssetManager/UIAssetMerger/StylesheetUIAssetMerger.php Normal file
View file

@ -0,0 +1,272 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\AssetManager\UIAssetMerger;
use Exception;
use lessc;
use Piwik\AssetManager\UIAsset;
use Piwik\AssetManager\UIAssetMerger;
use Piwik\Common;
use Piwik\Container\StaticContainer;
use Piwik\Exception\StylesheetLessCompileException;
use Piwik\Piwik;
use Piwik\Plugin\Manager;
class StylesheetUIAssetMerger extends UIAssetMerger
{
/**
* @var lessc
*/
private $lessCompiler;
/**
* @var UIAsset[]
*/
private $cssAssetsToReplace = array();
public function __construct($mergedAsset, $assetFetcher, $cacheBuster)
{
parent::__construct($mergedAsset, $assetFetcher, $cacheBuster);
$this->lessCompiler = self::getLessCompiler();
}
protected function getMergedAssets()
{
// note: we're using setImportDir on purpose (not addImportDir)
$this->lessCompiler->setImportDir(PIWIK_DOCUMENT_ROOT);
$concatenatedAssets = $this->getConcatenatedAssets();
$this->lessCompiler->setFormatter('classic');
try {
$compiled = $this->lessCompiler->compile($concatenatedAssets);
} catch(\Exception $e) {
// save the concated less files so we can debug the issue
$this->saveConcatenatedAssets($concatenatedAssets);
throw new StylesheetLessCompileException($e->getMessage());
}
foreach ($this->cssAssetsToReplace as $asset) {
// to fix #10173
$cssPath = $asset->getAbsoluteLocation();
$cssContent = $this->processFileContent($asset);
$compiled = str_replace($this->getCssStatementForReplacement($cssPath), $cssContent, $compiled);
}
$this->mergedContent = $compiled;
$this->cssAssetsToReplace = array();
return $compiled;
}
private function getCssStatementForReplacement($path)
{
return ".nonExistingSelectorOnlyForReplacementOfCssFiles {\n display: \"" . $path . "\";\n}";
}
protected function concatenateAssets()
{
$concatenatedContent = '';
foreach ($this->getAssetCatalog()->getAssets() as $uiAsset) {
$uiAsset->validateFile();
try {
$path = $uiAsset->getAbsoluteLocation();
} catch (Exception $e) {
$path = null;
}
if (!empty($path) && Common::stringEndsWith($path, '.css')) {
// to fix #10173
$concatenatedContent .= "\n" . $this->getCssStatementForReplacement($path) . "\n";
$this->cssAssetsToReplace[] = $uiAsset;
} else {
$content = $this->processFileContent($uiAsset);
$concatenatedContent .= $this->getFileSeparator() . $content;
}
}
/**
* Triggered after all less stylesheets are concatenated into one long string but before it is
* minified and merged into one file.
*
* This event can be used to add less stylesheets that are not located in a file on the disc.
*
* @param string $concatenatedContent The content of all concatenated less files.
*/
Piwik::postEvent('AssetManager.addStylesheets', array(&$concatenatedContent));
$this->mergedContent = $concatenatedContent;
}
/**
* @return lessc
* @throws Exception
*/
private static function getLessCompiler()
{
if (!class_exists("lessc")) {
throw new Exception("Less was added to composer during 2.0. ==> Execute this command to update composer packages: \$ php composer.phar install");
}
$less = new lessc();
return $less;
}
protected function generateCacheBuster()
{
$fileHash = $this->cacheBuster->md5BasedCacheBuster($this->getConcatenatedAssets());
return "/* compile_me_once=$fileHash */";
}
protected function getPreamble()
{
return $this->getCacheBusterValue() . "\n"
. "/* Matomo CSS file is compiled with Less. You may be interested in writing a custom Theme for Matomo! */\n";
}
protected function postEvent(&$mergedContent)
{
/**
* Triggered after all less stylesheets are compiled to CSS, minified and merged into
* one file, but before the generated CSS is written to disk.
*
* This event can be used to modify merged CSS.
*
* @param string $mergedContent The merged and minified CSS.
*/
Piwik::postEvent('AssetManager.filterMergedStylesheets', array(&$mergedContent));
}
public function getFileSeparator()
{
return '';
}
protected function processFileContent($uiAsset)
{
$pathsRewriter = $this->getCssPathsRewriter($uiAsset);
$content = $uiAsset->getContent();
$content = $this->rewriteCssImagePaths($content, $pathsRewriter);
$content = $this->rewriteCssImportPaths($content, $pathsRewriter);
return $content;
}
/**
* Rewrite CSS url() directives
*
* @param string $content
* @param callable $pathsRewriter
* @return string
*/
private function rewriteCssImagePaths($content, $pathsRewriter)
{
$content = preg_replace_callback("/(url\(['\"]?)([^'\")]*)/", $pathsRewriter, $content);
return $content;
}
/**
* Rewrite CSS import directives
*
* @param string $content
* @param callable $pathsRewriter
* @return string
*/
private function rewriteCssImportPaths($content, $pathsRewriter)
{
$content = preg_replace_callback("/(@import \")([^\")]*)/", $pathsRewriter, $content);
return $content;
}
/**
* Rewrite CSS url directives
* - rewrites paths defined relatively to their css/less definition file
* - rewrite windows directory separator \\ to /
*
* @param UIAsset $uiAsset
* @return \Closure
*/
private function getCssPathsRewriter($uiAsset)
{
$baseDirectory = dirname($uiAsset->getRelativeLocation());
$webDirs = Manager::getAlternativeWebRootDirectories();
return function ($matches) use ($baseDirectory, $webDirs) {
$absolutePath = PIWIK_DOCUMENT_ROOT . "/$baseDirectory/" . $matches[2];
// Allow to import extension less file
if (strpos($matches[2], '.') === false) {
$absolutePath .= '.less';
}
// Prevent from rewriting full path
$absolutePathReal = realpath($absolutePath);
if ($absolutePathReal) {
$relativePath = $baseDirectory . "/" . $matches[2];
$relativePath = str_replace('\\', '/', $relativePath);
$publicPath = $matches[1] . $relativePath;
} else {
foreach ($webDirs as $absPath => $relativePath) {
if (strpos($baseDirectory, $relativePath) === 0) {
if (strpos($matches[2], '.') === 0) {
// eg ../images/ok.png
$fileRelative = $baseDirectory . '/' . $matches[2];
$fileAbsolute = $absPath . str_replace($relativePath, '', $fileRelative);
if (file_exists($fileAbsolute)) {
return $matches[1] . $fileRelative;
}
} elseif (strpos($matches[2], 'plugins/') === 0) {
// eg plugins/Foo/images/ok.png
$fileRelative = substr($matches[2], strlen('plugins/'));
$fileAbsolute = $absPath . $fileRelative;
if (file_exists($fileAbsolute)) {
return $matches[1] . $relativePath . $fileRelative;
}
} elseif ($matches[1] === '@import "') {
$fileRelative = $baseDirectory . '/' . $matches[2];
$fileAbsolute = $absPath . str_replace($relativePath, '', $fileRelative);
if (file_exists($fileAbsolute)) {
return $matches[1] . $baseDirectory . '/' . $matches[2];
}
}
}
}
$publicPath = $matches[1] . $matches[2];
}
return $publicPath;
};
}
/**
* @param UIAsset $uiAsset
* @return int
*/
protected function countDirectoriesInPathToRoot($uiAsset)
{
$rootDirectory = realpath($uiAsset->getBaseDirectory());
if ($rootDirectory != PATH_SEPARATOR
&& substr($rootDirectory, -strlen(PATH_SEPARATOR)) !== PATH_SEPARATOR) {
$rootDirectory .= PATH_SEPARATOR;
}
$rootDirectoryLen = strlen($rootDirectory);
return $rootDirectoryLen;
}
private function saveConcatenatedAssets($concatenatedAssets)
{
$file = StaticContainer::get('path.tmp') . '/assets/uimergedassets.concat.less';
if (is_writable(dirname($file))) {
file_put_contents($file, $concatenatedAssets);
}
}
}

66
matomo/core/AssetManager/UIAssetMinifier.php Normal file
View file

@ -0,0 +1,66 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
* @method static \Piwik\AssetManager\UIAssetMinifier getInstance()
*/
namespace Piwik\AssetManager;
use Exception;
use JShrink\Minifier;
use Piwik\Singleton;
class UIAssetMinifier extends Singleton
{
const MINIFIED_JS_RATIO = 100;
protected function __construct()
{
self::validateDependency();
parent::__construct();
}
/**
* Indicates if the provided JavaScript content has already been minified or not.
* The heuristic is based on a custom ratio : (size of file) / (number of lines).
* The threshold (100) has been found empirically on existing files :
* - the ratio never exceeds 50 for non-minified content and
* - it never goes under 150 for minified content.
*
* @param string $content Contents of the JavaScript file
* @return boolean
*/
public function isMinifiedJs($content)
{
$lineCount = substr_count($content, "\n");
if ($lineCount == 0) {
return true;
}
$contentSize = strlen($content);
$ratio = $contentSize / $lineCount;
return $ratio > self::MINIFIED_JS_RATIO;
}
/**
* @param string $content
* @return string
*/
public function minifyJs($content)
{
return Minifier::minify($content);
}
private static function validateDependency()
{
if (!class_exists("JShrink\\Minifier")) {
throw new Exception("JShrink could not be found, maybe you are using Matomo from git and need to update Composer. $ php composer.phar update");
}
}
}

122
matomo/core/Auth.php Normal file
View file

@ -0,0 +1,122 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
use Exception;
/**
* Base interface for authentication implementations.
*
* Plugins that provide Auth implementations must provide a class that implements
* this interface. Additionally, an instance of that class must be set in the
* container with the 'Piwik\Auth' key during the
* [Request.initAuthenticationObject](https://developer.matomo.org/api-reference/events#requestinitauthenticationobject)
* event.
*
* Authentication implementations must support authentication via username and
* clear-text password and authentication via username and token auth. They can
* additionally support authentication via username and an MD5 hash of a password. If
* they don't support it, then [formless authentication](https://matomo.org/faq/how-to/faq_30/) will fail.
*
* Derived implementations should favor authenticating by password over authenticating
* by token auth. That is to say, if a token auth and a password are set, password
* authentication should be used.
*
* ### Examples
*
* **How an Auth implementation will be used**
*
* // authenticating by password
* $auth = StaticContainer::get('Piwik\Auth');
* $auth->setLogin('user');
* $auth->setPassword('password');
* $result = $auth->authenticate();
*
* // authenticating by token auth
* $auth = StaticContainer::get('Piwik\Auth');
* $auth->setLogin('user');
* $auth->setTokenAuth('...');
* $result = $auth->authenticate();
*
* @api
*/
interface Auth
{
/**
* Must return the Authentication module's name, e.g., `"Login"`.
*
* @return string
*/
public function getName();
/**
* Sets the authentication token to authenticate with.
*
* @param string $token_auth authentication token
*/
public function setTokenAuth($token_auth);
/**
* Returns the login of the user being authenticated.
*
* @return string
*/
public function getLogin();
/**
* Returns the secret used to calculate a user's token auth.
*
* A users token auth is generated using the user's login and this secret. The secret
* should be specific to the user and not easily guessed. Piwik's default Auth implementation
* uses an MD5 hash of a user's password.
*
* @return string
* @throws Exception if the token auth secret does not exist or cannot be obtained.
*/
public function getTokenAuthSecret();
/**
* Sets the login name to authenticate with.
*
* @param string $login The username.
*/
public function setLogin($login);
/**
* Sets the password to authenticate with.
*
* @param string $password Password (not hashed).
*/
public function setPassword($password);
/**
* Sets the hash of the password to authenticate with. The hash will be an MD5 hash.
*
* @param string $passwordHash The hashed password.
* @throws Exception if authentication by hashed password is not supported.
*/
public function setPasswordHash($passwordHash);
/**
* Authenticates a user using the login and password set using the setters. Can also authenticate
* via token auth if one is set and no password is set.
*
* Note: this method must successfully authenticate if the token auth supplied is a special hash
* of the user's real token auth. This is because the SessionInitializer class stores a
* hash of the token auth in the session cookie. You can calculate the token auth hash using the
* {@link \Piwik\Plugins\Login\SessionInitializer::getHashTokenAuth()} method.
*
* @return AuthResult
* @throws Exception if the Auth implementation has an invalid state (ie, no login
* was specified). Note: implementations are not **required** to throw
* exceptions for invalid state, but they are allowed to.
*/
public function authenticate();
}

120
matomo/core/Auth/Password.php Normal file
View file

@ -0,0 +1,120 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Auth;
use Exception;
use Piwik\Config;
/**
* Main class to handle actions related to password hashing and verification.
*
* @api
*/
class Password
{
/**
* Choose the used algorithm for password_hash depending on the config option
*
* @return string|int depending on PHP version
* @throws Exception
*/
private function preferredAlgorithm()
{
$passwordHashAlgorithm = Config::getInstance()->General['password_hash_algorithm'];
switch ($passwordHashAlgorithm) {
case "default":
return PASSWORD_DEFAULT;
case "bcrypt":
return PASSWORD_BCRYPT;
case "argon2i":
return PASSWORD_ARGON2I;
case "argon2id":
if (version_compare(PHP_VERSION, '7.3.0', '<')) {
throw new Exception("argon2id needs at leat PHP 7.3.0");
}
return PASSWORD_ARGON2ID;
default:
throw new Exception("invalid password_hash_algorithm");
}
}
/**
* Fetches argon2 options from config.ini.php
*
* @return array
*/
private function algorithmOptions()
{
$options = [];
$generalConfig = Config::getInstance()->General;
if ($generalConfig["password_hash_argon2_threads"] != "default") {
$options["threads"] = max($generalConfig["password_hash_argon2_threads"], 1);
}
if ($generalConfig["password_hash_argon2_memory_cost"] != "default") {
$options["memory_cost"] = max($generalConfig["password_hash_argon2_memory_cost"], 8 * $options["threads"]);
}
if ($generalConfig["password_hash_argon2_time_cost"] != "default") {
$options["time_cost"] = max($generalConfig["password_hash_argon2_time_cost"], 1);
}
return $options;
}
/**
* Hashes a password with the configured algorithm.
*
* @param string $password
* @return string
*/
public function hash($password)
{
return password_hash($password, $this->preferredAlgorithm(), $this->algorithmOptions());
}
/**
* Returns information about a hashed password (algo, options, ...).
*
* Can be used to verify whether a string is compatible with password_hash().
*
* @param string
* @return array
*/
public function info($hash)
{
return password_get_info($hash);
}
/**
* Rehashes a user's password if necessary.
*
* This method expects the password to be pre-hashed by
* \Piwik\Plugins\UsersManager\UsersManager::getPasswordHash().
*
* @param string $hash
* @return boolean
*/
public function needsRehash($hash)
{
return password_needs_rehash($hash, $this->preferredAlgorithm(), $this->algorithmOptions());
}
/**
* Verifies a user's password against the provided hash.
*
* This method expects the password to be pre-hashed by
* \Piwik\Plugins\UsersManager\UsersManager::getPasswordHash().
*
* @param string $password
* @param string $hash
* @return boolean
*/
public function verify($password, $hash)
{
return password_verify($password, $hash);
}
}

108
matomo/core/AuthResult.php Normal file
View file

@ -0,0 +1,108 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
/**
* Authentication result. This is what is returned by authentication attempts using {@link Auth}
* implementations.
*
* @api
*/
class AuthResult
{
const FAILURE = 0;
const SUCCESS = 1;
const SUCCESS_SUPERUSER_AUTH_CODE = 42;
/**
* token_auth parameter used to authenticate in the API
*
* @var string
*/
protected $tokenAuth = null;
/**
* The login used to authenticate.
*
* @var string
*/
protected $login = null;
/**
* The authentication result code. Can be self::FAILURE, self::SUCCESS, or
* self::SUCCESS_SUPERUSER_AUTH_CODE.
*
* @var int
*/
protected $code = null;
/**
* Constructor for AuthResult
*
* @param int $code
* @param string $login identity
* @param string $tokenAuth
*/
public function __construct($code, $login, $tokenAuth)
{
$this->code = (int)$code;
$this->login = $login;
$this->tokenAuth = $tokenAuth;
}
/**
* Returns the login used to authenticate.
*
* @return string
*/
public function getIdentity()
{
return $this->login;
}
/**
* Returns the token_auth to authenticate the current user in the API
*
* @return string
*/
public function getTokenAuth()
{
return $this->tokenAuth;
}
/**
* Returns the authentication result code.
*
* @return int
*/
public function getCode()
{
return $this->code;
}
/**
* Returns true if the user has Super User access, false otherwise.
*
* @return bool
*/
public function hasSuperUserAccess()
{
return $this->getCode() == self::SUCCESS_SUPERUSER_AUTH_CODE;
}
/**
* Returns true if this result was successfully authentication.
*
* @return bool
*/
public function wasAuthenticationSuccessful()
{
return $this->code > self::FAILURE;
}
}

65
matomo/core/BaseFactory.php Normal file
View file

@ -0,0 +1,65 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
use Exception;
/**
* Base class for all factory types.
*
* Factory types are base classes that contain a **factory** method. This method is used to instantiate
* concrete instances by a specified string ID. Fatal errors do not occur if a class does not exist.
* Instead an exception is thrown.
*
* Derived classes should override the **getClassNameFromClassId** and **getInvalidClassIdExceptionMessage**
* static methods.
*/
abstract class BaseFactory
{
/**
* Creates a new instance of a class using a string ID.
*
* @param string $classId The ID of the class.
* @return \Piwik\DataTable\Renderer
* @throws Exception if $classId is invalid.
*/
public static function factory($classId)
{
$className = static::getClassNameFromClassId($classId);
if (!class_exists($className)) {
self::sendPlainHeader();
throw new Exception(static::getInvalidClassIdExceptionMessage($classId));
}
return new $className;
}
private static function sendPlainHeader()
{
Common::sendHeader('Content-Type: text/plain; charset=utf-8');
}
/**
* Should return a class name based on the class's associated string ID.
*/
protected static function getClassNameFromClassId($id)
{
return $id;
}
/**
* Should return a message to use in an Exception when an invalid class ID is supplied to
* {@link factory()}.
*/
protected static function getInvalidClassIdExceptionMessage($id)
{
return "Invalid class ID '$id' for " . get_called_class() . "::factory().";
}
}

116
matomo/core/Cache.php Normal file
View file

@ -0,0 +1,116 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
use Piwik\Container\StaticContainer;
class Cache
{
/**
* This can be considered as the default cache to use in case you don't know which one to pick. It does not support
* the caching of any objects though. Only boolean, numbers, strings and arrays are supported. Whenever you request
* an entry from the cache it will fetch the entry. Cache entries might be persisted but not necessarily. It
* depends on the configured backend.
*
* @return \Matomo\Cache\Lazy
*/
public static function getLazyCache()
{
return StaticContainer::get('Matomo\Cache\Lazy');
}
/**
* This class is used to cache any data during one request. It won't be persisted between requests and it can
* cache all kind of data, even objects or resources. This cache is very fast.
*
* @return \Matomo\Cache\Transient
*/
public static function getTransientCache()
{
return StaticContainer::get('Matomo\Cache\Transient');
}
/**
* This cache stores all its cache entries under one "cache" entry in a configurable backend.
*
* This comes handy for things that you need very often, nearly in every request. For example plugin metadata, the
* list of tracker plugins, the list of available languages, ...
* Instead of having to read eg. a hundred cache entries from files (or any other backend) it only loads one cache
* entry which contains the hundred keys. Should be used only for things that you need very often and only for
* cache entries that are not too large to keep loading and parsing the single cache entry fast.
* All cache entries it contains have the same life time. For fast performance it won't validate any cache ids.
* It is not possible to cache any objects using this cache.
*
* @return \Matomo\Cache\Eager
*/
public static function getEagerCache()
{
return StaticContainer::get('Matomo\Cache\Eager');
}
public static function flushAll()
{
self::getLazyCache()->flushAll();
self::getTransientCache()->flushAll();
self::getEagerCache()->flushAll();
}
/**
* @param $type
* @return \Matomo\Cache\Backend
*/
public static function buildBackend($type)
{
$factory = new \Matomo\Cache\Backend\Factory();
$options = self::getOptions($type);
$backend = $factory->buildBackend($type, $options);
return $backend;
}
private static function getOptions($type)
{
$options = self::getBackendOptions($type);
switch ($type) {
case 'file':
$options = array('directory' => StaticContainer::get('path.cache'));
break;
case 'chained':
foreach ($options['backends'] as $backend) {
$options[$backend] = self::getOptions($backend);
}
break;
case 'redis':
if (!empty($options['timeout'])) {
$options['timeout'] = (float)Common::forceDotAsSeparatorForDecimalPoint($options['timeout']);
}
break;
}
return $options;
}
private static function getBackendOptions($backend)
{
$key = ucfirst($backend) . 'Cache';
$options = Config::getInstance()->$key;
return $options;
}
}

88
matomo/core/CacheId.php Normal file
View file

@ -0,0 +1,88 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik;
use Piwik\Container\StaticContainer;
use Piwik\Plugin\Manager;
class CacheId
{
public static function languageAware($cacheId)
{
return $cacheId . '-' . StaticContainer::get('Piwik\Translation\Translator')->getCurrentLanguage();
}
public static function pluginAware($cacheId)
{
$pluginManager = Manager::getInstance();
$pluginNames = $pluginManager->getLoadedPluginsName();
$cacheId = $cacheId . '-' . md5(implode('', $pluginNames));
$cacheId = self::languageAware($cacheId);
return $cacheId;
}
public static function siteAware($cacheId, array $idSites = null)
{
if ($idSites === null) {
$idSites = self::getIdSiteList('idSite');
$cacheId .= self::idSiteListCacheKey($idSites);
$idSites = self::getIdSiteList('idSites');
$cacheId .= self::idSiteListCacheKey($idSites);
$idSites = self::getIdSiteList('idsite'); // tracker param
$cacheId .= self::idSiteListCacheKey($idSites);
} else {
$cacheId .= self::idSiteListCacheKey($idSites);
}
return $cacheId;
}
private static function getIdSiteList($queryParamName)
{
if (empty($_GET[$queryParamName])
&& empty($_POST[$queryParamName])
) {
return [];
}
$idSiteGetParam = [];
if (!empty($_GET[$queryParamName])) {
$value = $_GET[$queryParamName];
$idSiteGetParam = is_array($value) ? $value : explode(',', $value);
}
$idSitePostParam = [];
if (!empty($_POST[$queryParamName])) {
$value = $_POST[$queryParamName];
$idSitePostParam = is_array($value) ? $value : explode(',', $value);
}
$idSiteList = array_merge($idSiteGetParam, $idSitePostParam);
$idSiteList = array_map('intval', $idSiteList);
$idSiteList = array_unique($idSiteList);
sort($idSiteList);
return $idSiteList;
}
private static function idSiteListCacheKey($idSites)
{
if (empty($idSites)) {
return '';
}
if (count($idSites) <= 5) {
return '-' . implode('_', $idSites); // we keep the cache key readable when possible
} else {
return '-' . md5(implode('_', $idSites)); // we need to shorten it
}
}
}

132
matomo/core/Category/Category.php Normal file
View file

@ -0,0 +1,132 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Category;
use Piwik\Piwik;
/**
* Base type for category. lets you change the name for a categoryId and specify a different order
* so the category appears eg at a different order in the reporting menu.
*
* This class is for now not exposed as public API until needed. Categories of plugins will be automatically
* displayed in the menu at the very right after all core categories.
*/
class Category
{
/**
* The id of the category as specified eg in {@link Piwik\Widget\WidgetConfig::setCategoryId()`} or
* {@link Piwik\Report\getCategoryId()}. The id is used as the name in the menu and will be visible in the
* URL.
*
* @var string Should be a translation key, eg 'General_Vists'
*/
protected $id = '';
/**
* @var Subcategory[]
*/
protected $subcategories = array();
/**
* The order of the category. The lower the value the further left the category will appear in the menu.
* @var int
*/
protected $order = 99;
/**
* The icon for this category, eg 'icon-user'
* @var int
*/
protected $icon = '';
/**
* @param int $order
* @return static
*/
public function setOrder($order)
{
$this->order = (int) $order;
return $this;
}
public function getOrder()
{
return $this->order;
}
public function setId($id)
{
$this->id = $id;
return $this;
}
public function getId()
{
return $this->id;
}
public function getDisplayName()
{
return Piwik::translate($this->getId());
}
public function addSubcategory(Subcategory $subcategory)
{
$subcategoryId = $subcategory->getId();
if ($this->hasSubcategory($subcategoryId)) {
throw new \Exception(sprintf('Subcategory %s already exists for category %s', $subcategoryId, $this->getId()));
}
$this->subcategories[$subcategoryId] = $subcategory;
}
public function hasSubcategory($subcategoryId)
{
return isset($this->subcategories[$subcategoryId]);
}
public function getSubcategory($subcategoryId)
{
if ($this->hasSubcategory($subcategoryId)) {
return $this->subcategories[$subcategoryId];
}
}
/**
* @return Subcategory[]
*/
public function getSubcategories()
{
return array_values($this->subcategories);
}
public function hasSubCategories()
{
return !empty($this->subcategories);
}
public function setIcon($icon)
{
$this->icon = $icon;
return $this;
}
public function getIcon()
{
return $this->icon;
}
/**
* Get the help text (if any) for this category.
* @return null
*/
public function getHelp()
{
return null;
}
}

94
matomo/core/Category/CategoryList.php Normal file
View file

@ -0,0 +1,94 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Category;
use Piwik\Container\StaticContainer;
/**
* Base type for category. lets you change the name for a categoryId and specify a different order
* so the category appears eg at a different order in the reporting menu.
*
* This class is for now not exposed as public API until needed. Categories of plugins will be automatically
* displayed in the menu at the very right after all core categories.
*/
class CategoryList
{
/**
* @var Category[] indexed by categoryId
*/
private $categories = array();
public function addCategory(Category $category)
{
$categoryId = $category->getId();
if ($this->hasCategory($categoryId)) {
throw new \Exception(sprintf('Category %s already exists', $categoryId));
}
$this->categories[$categoryId] = $category;
}
public function getCategories()
{
return $this->categories;
}
public function hasCategory($categoryId)
{
return isset($this->categories[$categoryId]);
}
/**
* Get the category having the given id, if possible.
*
* @param string $categoryId
* @return Category|null
*/
public function getCategory($categoryId)
{
if ($this->hasCategory($categoryId)) {
return $this->categories[$categoryId];
}
}
/**
* @return CategoryList
*/
public static function get()
{
$list = new CategoryList();
$categories = StaticContainer::get('Piwik\Plugin\Categories');
foreach ($categories->getAllCategories() as $category) {
$list->addCategory($category);
}
// move subcategories into categories
foreach ($categories->getAllSubcategories() as $subcategory) {
$categoryId = $subcategory->getCategoryId();
if (!$categoryId) {
continue;
}
if ($list->hasCategory($categoryId)) {
$category = $list->getCategory($categoryId);
} else {
$category = new Category();
$category->setId($categoryId);
$list->addCategory($category);
}
$category->addSubcategory($subcategory);
}
return $list;
}
}

155
matomo/core/Category/Subcategory.php Normal file
View file

@ -0,0 +1,155 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\Category;
/**
* Base type for subcategories.
*
* All widgets within a subcategory will be rendered in the Piwik reporting UI under the same page. By default
* you do not have to specify any subcategory as they are created automatically. Only create a subcategory if you
* want to change the name for a specific subcategoryId or if you want to specify a different order so the subcategory
* appears eg at a different order in the reporting menu. It also affects the order of reports in
* `API.getReportMetadata` and wherever we display any reports.
*
* To define a subcategory just place a subclass within the `Categories` folder of your plugin.
*
* Subcategories can also be added through the {@hook Subcategory.addSubcategories} event.
*
* @api since Piwik 3.0.0
*/
class Subcategory
{
/**
* The id of the subcategory, see eg {@link Piwik\Widget\WidgetConfig::setSubcategoryId()`} or
* {@link Piwik\Report\getSubcategoryId()}. The id will be used in the Piwik reporting URL and as the name
* in the Piwik reporting submenu. If you want to define a different URL and name, specify a {@link $name}.
* For example you might want to have the actual GoalId (eg '4') in the URL but the actual goal name in the
* submenu (eg 'Downloads'). In this case one should specify `$id=4;$name='Downloads'`.
*
* @var string eg 'General_Overview' or 'VisitTime_ByServerTimeWidgetName'.
*/
protected $id = '';
/**
* The id of the category the subcategory belongs to, must be specified.
* See {@link Piwik\Widget\WidgetConfig::setCategoryId()`} or {@link Piwik\Report\getCategoryId()}.
*
* @var string A translation key eg 'General_Visits' or 'Goals_Goals'
*/
protected $categoryId = '';
/**
* The name that shall be used in the menu etc, defaults to the specified {@link $id}. See {@link $id}.
* @var string
*/
protected $name = '';
/**
* The order of the subcategory. The lower the value the earlier a widget or a report will be displayed.
* @var int
*/
protected $order = 99;
/**
* Sets (overwrites) the id of the subcategory see {@link $id}.
*
* @param string $id A translation key eg 'General_Overview'.
* @return static
*/
public function setId($id)
{
$this->id = $id;
return $this;
}
/**
* Get the id of the subcategory.
* @return string
*/
public function getId()
{
return $this->id;
}
/**
* Get the specified categoryId see {@link $categoryId}.
*
* @return string
*/
public function getCategoryId()
{
return $this->categoryId;
}
/**
* Sets (overwrites) the categoryId see {@link $categoryId}.
*
* @param string $categoryId
* @return static
*/
public function setCategoryId($categoryId)
{
$this->categoryId = $categoryId;
return $this;
}
/**
* Sets (overwrites) the name see {@link $name} and {@link $id}.
*
* @param string $name A translation key eg 'General_Overview'.
* @return static
*/
public function setName($name)
{
$this->name = $name;
return $this;
}
/**
* Get the name of the subcategory.
* @return string
*/
public function getName()
{
if (!empty($this->name)) {
return $this->name;
}
return $this->id;
}
/**
* Sets (overwrites) the order see {@link $order}.
*
* @param int $order
* @return static
*/
public function setOrder($order)
{
$this->order = (int) $order;
return $this;
}
/**
* Get the order of the subcategory.
* @return int
*/
public function getOrder()
{
return $this->order;
}
/**
* Get the help text (if any) for this category.
* @return null
*/
public function getHelp()
{
return null;
}
}

218
matomo/core/Changes/Model.php Normal file
View file

@ -0,0 +1,218 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Changes;
use Piwik\Piwik;
use Piwik\Common;
use Piwik\Date;
use Piwik\Db;
use Piwik\Tracker\Db\DbException;
use Piwik\Updater\Migration;
use Piwik\Container\StaticContainer;
use Piwik\Plugin\Manager as PluginManager;
/**
* Change model class
*
* Handle all data access operations for changes
*
*/
class Model
{
const NO_CHANGES_EXIST = 0;
const CHANGES_EXIST = 1;
const NEW_CHANGES_EXIST = 2;
private $pluginManager;
/**
* @var Db\AdapterInterface
*/
private $db;
/**
* @param Db\AdapterInterface|null $db
* @param PluginManager|null $pluginManager
*/
public function __construct(?Db\AdapterInterface $db = null, ?PluginManager $pluginManager = null)
{
$this->db = ($db ?? Db::get());
$this->pluginManager = ($pluginManager ?? PluginManager::getInstance());
}
/**
* Add any new changes for a plugin to the changes table
*
* @param string $pluginName
*
* @throws \Exception
*/
public function addChanges(string $pluginName): void
{
if ($this->pluginManager->isValidPluginName($pluginName) && $this->pluginManager->isPluginInFilesystem($pluginName)) {
$plugin = $this->pluginManager->loadPlugin($pluginName);
if (!$plugin) {
return;
}
$changes = $plugin->getChanges();
foreach ($changes as $change) {
$this->addChange($pluginName, $change);
}
}
}
/**
* Remove all changes for a plugin
*
* @param string $pluginName
*/
public function removeChanges(string $pluginName): void
{
$table = Common::prefixTable('changes');
try {
$this->db->query("DELETE FROM " . $table . " WHERE plugin_name = ?", [$pluginName]);
} catch (\Exception $e) {
if (Db::get()->isErrNo($e, Migration\Db::ERROR_CODE_TABLE_NOT_EXISTS)) {
return;
}
throw $e;
}
}
/**
* Add a change item to the database table
*
* @param string $pluginName
* @param array $change
*/
public function addChange(string $pluginName, array $change): void
{
if(!isset($change['version']) || !isset($change['title']) || !isset($change['description'])) {
StaticContainer::get('Psr\Log\LoggerInterface')->warning(
"Change item for plugin {plugin} missing version, title or description fields - ignored",
['plugin' => $pluginName]);
return;
}
$table = Common::prefixTable('changes');
$fields = ['created_time', 'plugin_name', 'version', 'title', 'description'];
$params = [Date::now()->getDatetime(), $pluginName, $change['version'], $change['title'], $change['description']];
if (isset($change['link_name']) && isset($change['link'])) {
$fields[] = 'link_name';
$fields[] = 'link';
$params[] = $change['link_name'];
$params[] = $change['link'];
}
$insertSql = 'INSERT IGNORE INTO ' . $table . ' ('.implode(',', $fields).')
VALUES ('.Common::getSqlStringFieldsArray($params).')';
try {
$this->db->query($insertSql, $params);
} catch (\Exception $e) {
if (Db::get()->isErrNo($e, Migration\Db::ERROR_CODE_TABLE_NOT_EXISTS)) {
return;
}
throw $e;
}
}
/**
* Check if any changes items exist
*
* @param int|null $newerThanId Only count new changes as having a key > than this sequential key
*
* @return int
*/
public function doChangesExist(?int $newerThanId = null): int
{
$changes = $this->getChangeItems();
$all = 0;
$new = 0;
foreach ($changes as $c) {
$all++;
if ($newerThanId !== null && isset($c['idchange']) && $c['idchange'] > $newerThanId) {
$new++;
}
}
if ($all === 0) {
return self::NO_CHANGES_EXIST;
} else if ($all > 0 && $new === 0) {
return self::CHANGES_EXIST;
} else {
return self::NEW_CHANGES_EXIST;
}
}
/**
* Return an array of change items from the changes table
*
* @return array
* @throws DbException
*/
public function getChangeItems(): array
{
$showAtLeast = 10; // Always show at least this number of changes
$expireOlderThanDays = 90; // Don't show changes that were added to the table more than x days ago
$table = Common::prefixTable('changes');
$selectSql = "SELECT * FROM " . $table . " WHERE title IS NOT NULL ORDER BY idchange DESC";
try {
$changes = $this->db->fetchAll($selectSql);
} catch (\Exception $e) {
if (Db::get()->isErrNo($e, Migration\Db::ERROR_CODE_TABLE_NOT_EXISTS)) {
return [];
}
throw $e;
}
// Remove expired changes, only if there are at more than the minimum changes
$cutOffDate = Date::now()->subDay($expireOlderThanDays);
foreach ($changes as $k => $change) {
if (isset($change['idchange'])) {
$changes[$k]['idchange'] = (int)$change['idchange'];
}
if (count($changes) > $showAtLeast && $change['created_time'] < $cutOffDate) {
unset($changes[$k]);
}
}
/**
* Event triggered before changes are displayed
*
* Can be used to filter out unwanted changes
*
* **Example**
*
* Piwik::addAction('Changes.filterChanges', function ($changes) {
* foreach ($changes as $k => $c) {
* // Hide changes for the CoreHome plugin
* if (isset($c['plugin_name']) && $c['plugin_name'] == 'CoreHome') {
* unset($changes[$k]);
* }
* }
* });
*
* @param array &$changes
*/
Piwik::postEvent('Changes.filterChanges', array(&$changes));
return $changes;
}
}

77
matomo/core/Changes/UserChanges.php Normal file
View file

@ -0,0 +1,77 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Changes;
use Piwik\Db;
use Piwik\Changes\Model as ChangesModel;
use Piwik\Plugins\UsersManager\Model as UsersModel;
/**
* CoreHome user changes class
*/
class UserChanges
{
/**
* @var Db\AdapterInterface
*/
private $db;
private $user;
/**
* @param array $user
* @param Db\AdapterInterface|null $db
*/
public function __construct(array $user, ?Db\AdapterInterface $db = null)
{
$this->db = ($db ?? Db::get());
$this->user = $user;
}
/**
* Return a value indicating if there are any changes available to show the user
*
* @return int Changes\Model::NO_CHANGES_EXIST, Changes\Model::CHANGES_EXIST or Changes\Model::NEW_CHANGES_EXIST
* @throws \Exception
*/
public function getNewChangesStatus(): int
{
$idchangeLastViewed = (isset($this->user['idchange_last_viewed']) ? $this->user['idchange_last_viewed'] : null);
$changesModel = new ChangesModel($this->db);
return $changesModel->doChangesExist($idchangeLastViewed);
}
/**
* Return an array of changes and update the user's changes last viewed value
*
* @return array
*/
public function getChanges(): array
{
$changesModel = new ChangesModel(Db::get());
$changes = $changesModel->getChangeItems();
// Record the time that changes were viewed for the current user
$maxId = null;
foreach ($changes as $k => $change) {
if ($maxId < $change['idchange']) {
$maxId = $change['idchange'];
}
}
if ($maxId) {
$usersModel = new UsersModel();
$usersModel->updateUserFields($this->user['login'], ['idchange_last_viewed' => $maxId]);
}
return $changes;
}
}

505
matomo/core/CliMulti.php Normal file
View file

@ -0,0 +1,505 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik;
use Piwik\Archiver\Request;
use Piwik\CliMulti\CliPhp;
use Piwik\CliMulti\Output;
use Piwik\CliMulti\OutputInterface;
use Piwik\CliMulti\Process;
use Piwik\CliMulti\StaticOutput;
use Piwik\Container\StaticContainer;
use Psr\Log\LoggerInterface;
use Psr\Log\NullLogger;
/**
* Class CliMulti.
*/
class CliMulti
{
const BASE_WAIT_TIME = 250000; // 250 * 1000 = 250ms
/**
* If set to true or false it will overwrite whether async is supported or not.
*
* @var null|bool
*/
public $supportsAsync = null;
/**
* @var Process[]
*/
private $processes = array();
/**
* If set it will issue at most concurrentProcessesLimit requests
* @var int
*/
private $concurrentProcessesLimit = null;
/**
* @var OutputInterface[]
*/
private $outputs = array();
private $acceptInvalidSSLCertificate = false;
/**
* @var bool
*/
private $runAsSuperUser = false;
/**
* Only used when doing synchronous curl requests.
*
* @var string
*/
private $urlToPiwik = null;
private $phpCliOptions = '';
/**
* @var callable
*/
private $onProcessFinish = null;
/**
* @var Timer[]
*/
protected $timers = [];
protected $isTimingRequests = false;
/**
* @var LoggerInterface
*/
private $logger;
public function __construct(LoggerInterface $logger = null)
{
$this->supportsAsync = $this->supportsAsync();
$this->logger = $logger ?: new NullLogger();
}
/**
* It will request all given URLs in parallel (async) using the CLI and wait until all requests are finished.
* If multi cli is not supported (eg windows) it will initiate an HTTP request instead (not async).
*
* @param string[] $piwikUrls An array of urls, for instance:
*
* `array('http://www.example.com/piwik?module=API...')`
*
* **Make sure query parameter values are properly encoded in the URLs.**
*
* @return array The response of each URL in the same order as the URLs. The array can contain null values in case
* there was a problem with a request, for instance if the process died unexpected.
*/
public function request(array $piwikUrls)
{
if ($this->isTimingRequests) {
foreach ($piwikUrls as $url) {
$this->timers[] = new Timer();
}
}
$chunks = array($piwikUrls);
if ($this->concurrentProcessesLimit) {
$chunks = array_chunk($piwikUrls, $this->concurrentProcessesLimit);
}
$results = array();
foreach ($chunks as $urlsChunk) {
$results = array_merge($results, $this->requestUrls($urlsChunk));
}
return $results;
}
/**
* Forwards the given configuration options to the PHP cli command.
* @param string $phpCliOptions eg "-d memory_limit=8G -c=path/to/php.ini"
*/
public function setPhpCliConfigurationOptions($phpCliOptions)
{
$this->phpCliOptions = (string) $phpCliOptions;
}
/**
* Ok, this sounds weird. Why should we care about ssl certificates when we are in CLI mode? It is needed for
* our simple fallback mode for Windows where we initiate HTTP requests instead of CLI.
* @param $acceptInvalidSSLCertificate
*/
public function setAcceptInvalidSSLCertificate($acceptInvalidSSLCertificate)
{
$this->acceptInvalidSSLCertificate = $acceptInvalidSSLCertificate;
}
/**
* @param $limit int Maximum count of requests to issue in parallel
*/
public function setConcurrentProcessesLimit($limit)
{
$this->concurrentProcessesLimit = $limit;
}
public function runAsSuperUser($runAsSuperUser = true)
{
$this->runAsSuperUser = $runAsSuperUser;
}
private function start($piwikUrls)
{
$numUrls = count($piwikUrls);
foreach ($piwikUrls as $index => $url) {
$shouldStart = null;
if ($url instanceof Request) {
$shouldStart = $url->start();
}
$cmdId = $this->generateCommandId($url) . $index;
if ($shouldStart === Request::ABORT) {
// output is needed to ensure same order of url to response
$output = new StaticOutput($cmdId);
$output->write(serialize(array('aborted' => '1')));
$this->outputs[] = $output;
} else {
$this->executeUrlCommand($cmdId, $url, $numUrls);
}
}
}
private function executeUrlCommand($cmdId, $url, $numUrls)
{
if ($this->supportsAsync) {
if ($numUrls === 1) {
$output = new StaticOutput($cmdId);
$this->executeSyncCli($url, $output);
} else {
$output = new Output($cmdId);
$this->executeAsyncCli($url, $output, $cmdId);
}
} else {
$output = new StaticOutput($cmdId);
$this->executeNotAsyncHttp($url, $output);
}
$this->outputs[] = $output;
}
private function buildCommand($hostname, $query, $outputFileIfAsync, $doEsacpeArg = true)
{
$bin = $this->findPhpBinary();
$superuserCommand = $this->runAsSuperUser ? "--superuser" : "";
$append = '2>&1';
if ($outputFileIfAsync) {
$append = sprintf(' > %s 2>&1 &', $outputFileIfAsync);
}
if ($doEsacpeArg) {
$hostname = escapeshellarg($hostname);
$query = escapeshellarg($query);
}
return sprintf('%s %s %s/console climulti:request -q --matomo-domain=%s %s %s %s',
$bin, $this->phpCliOptions, PIWIK_INCLUDE_PATH, $hostname, $superuserCommand, $query,$append);
}
private function getResponse()
{
$response = array();
foreach ($this->outputs as $output) {
$content = $output->get();
// Remove output that can be ignored in climulti . works around some worpdress setups where the hash bang may
// be printed
$search = '#!/usr/bin/env php';
if (!empty($content)
&& is_string($content)
&& mb_substr(trim($content), 0, strlen($search)) === $search) {
$content = trim(mb_substr(trim($content), strlen($search)));
}
$response[] = $content;
}
return $response;
}
private function hasFinished()
{
foreach ($this->processes as $index => $process) {
$hasStarted = $process->hasStarted();
if (!$hasStarted && 8 <= $process->getSecondsSinceCreation()) {
// if process was created more than 8 seconds ago but still not started there must be something wrong.
// ==> declare the process as finished
$process->finishProcess();
continue;
} elseif (!$hasStarted) {
return false;
}
if ($process->isRunning()) {
return false;
}
$pid = $process->getPid();
foreach ($this->outputs as $output) {
if ($output->getOutputId() === $pid && $output->isAbnormal()) {
$process->finishProcess();
continue;
}
}
if ($process->hasFinished()) {
// prevent from checking this process over and over again
unset($this->processes[$index]);
if ($this->isTimingRequests) {
$this->timers[$index]->finish();
}
if ($this->onProcessFinish) {
$onProcessFinish = $this->onProcessFinish;
$onProcessFinish($pid);
}
}
}
return true;
}
private function generateCommandId($command)
{
return substr(Common::hash($command . microtime(true) . rand(0, 99999)), 0, 100);
}
/**
* What is missing under windows? Detection whether a process is still running in Process::isProcessStillRunning
* and how to send a process into background in start()
*/
public function supportsAsync()
{
$supportsAsync = Process::isSupported() && !Common::isPhpCgiType() && $this->findPhpBinary();
/**
* Triggered to allow plugins to force the usage of async cli multi execution or to disable it.
*
* **Example**
*
* public function supportsAsync(&$supportsAsync)
* {
* $supportsAsync = false; // do not allow async climulti execution
* }
*
* @param bool &$supportsAsync Whether async is supported or not.
*/
Piwik::postEvent('CliMulti.supportsAsync', array(&$supportsAsync));
return $supportsAsync;
}
private function findPhpBinary()
{
$cliPhp = new CliPhp();
return $cliPhp->findPhpBinary();
}
private function cleanup()
{
foreach ($this->processes as $pid) {
$pid->finishProcess();
}
foreach ($this->outputs as $output) {
$output->destroy();
}
$this->processes = array();
$this->outputs = array();
}
/**
* Remove files older than one week. They should be cleaned up automatically after each request but for whatever
* reason there can be always some files left.
*/
public static function cleanupNotRemovedFiles()
{
$timeOneWeekAgo = strtotime('-1 week');
$files = _glob(self::getTmpPath() . '/*');
if (empty($files)) {
return;
}
foreach ($files as $file) {
if (file_exists($file)) {
$timeLastModified = filemtime($file);
if ($timeLastModified !== false && $timeOneWeekAgo > $timeLastModified) {
unlink($file);
}
}
}
}
public static function getTmpPath()
{
return StaticContainer::get('path.tmp') . '/climulti';
}
private function executeAsyncCli($url, Output $output, $cmdId)
{
$this->processes[] = new Process($cmdId);
$url = $this->appendTestmodeParamToUrlIfNeeded($url);
$query = UrlHelper::getQueryFromUrl($url, array('pid' => $cmdId, 'runid' => getmypid()));
$hostname = Url::getHost($checkIfTrusted = false);
$command = $this->buildCommand($hostname, $query, $output->getPathToFile());
$this->logger->debug("Running command: {command}", ['command' => $command]);
shell_exec($command);
}
private function executeSyncCli($url, StaticOutput $output)
{
$url = $this->appendTestmodeParamToUrlIfNeeded($url);
$query = UrlHelper::getQueryFromUrl($url, array());
$hostname = Url::getHost($checkIfTrusted = false);
$command = $this->buildCommand($hostname, $query, '', true);
$this->logger->debug("Running command: {command}", ['command' => $command]);
$result = shell_exec($command);
if ($result) {
$result = trim($result);
}
$output->write($result);
}
private function executeNotAsyncHttp($url, StaticOutput $output)
{
$piwikUrl = $this->urlToPiwik ?: SettingsPiwik::getPiwikUrl();
if (empty($piwikUrl)) {
$piwikUrl = 'http://' . Url::getHost() . '/';
}
$url = $piwikUrl . $url;
if (Config::getInstance()->General['force_ssl'] == 1) {
$url = str_replace("http://", "https://", $url);
}
$requestBody = null;
if ($this->runAsSuperUser) {
$tokenAuth = self::getSuperUserTokenAuth();
if (strpos($url, '?') === false) {
$url .= '?';
} else {
$url .= '&';
}
$requestBody = 'token_auth=' . $tokenAuth;
}
try {
$this->logger->debug("Execute HTTP API request: " . $url);
$response = Http::sendHttpRequestBy('curl', $url, $timeout = 0, $userAgent = null, $destinationPath = null, $file = null, $followDepth = 0, $acceptLanguage = false, $this->acceptInvalidSSLCertificate, false, false, 'POST', null, null, $requestBody, [], $forcePost = true);
$output->write($response);
} catch (\Exception $e) {
$message = "Got invalid response from API request: $url. ";
if (isset($response) && empty($response)) {
$message .= "The response was empty. This usually means a server error. This solution to this error is generally to increase the value of 'memory_limit' in your php.ini file. Please check your Web server Error Log file for more details.";
} else {
$message .= "Response was '" . $e->getMessage() . "'";
}
$output->write($message);
$this->logger->debug($message, ['exception' => $e]);
}
}
private function appendTestmodeParamToUrlIfNeeded($url)
{
$isTestMode = defined('PIWIK_TEST_MODE');
if ($isTestMode && false === strpos($url, '?')) {
$url .= "?testmode=1";
} elseif ($isTestMode) {
$url .= "&testmode=1";
}
return $url;
}
/**
* @param array $piwikUrls
* @return array
*/
private function requestUrls(array $piwikUrls)
{
$this->start($piwikUrls);
$startTime = time();
do {
$elapsed = time() - $startTime;
$timeToWait = $this->getTimeToWaitBeforeNextCheck($elapsed);
if (count($this->processes)) {
usleep($timeToWait);
}
} while (!$this->hasFinished());
$results = $this->getResponse();
$this->cleanup();
self::cleanupNotRemovedFiles();
return $results;
}
private static function getSuperUserTokenAuth()
{
return Piwik::requestTemporarySystemAuthToken('CliMultiNonAsyncArchive', 36);
}
public function setUrlToPiwik($urlToPiwik)
{
$this->urlToPiwik = $urlToPiwik;
}
public function onProcessFinish(callable $callback)
{
$this->onProcessFinish = $callback;
}
// every minute that passes adds an extra 100ms to the wait time. so 5 minutes results in 500ms extra, 20mins results in 2s extra.
private function getTimeToWaitBeforeNextCheck($elapsed)
{
$minutes = floor($elapsed / 60);
return self::BASE_WAIT_TIME + $minutes * 100000; // 100 * 1000 = 100ms
}
public static function isCliMultiRequest()
{
return Common::getRequestVar('pid', false) !== false;
}
public function timeRequests()
{
$this->timers = [];
$this->isTimingRequests = true;
}
public function getTimers()
{
return $this->timers;
}
}

116
matomo/core/CliMulti/CliPhp.php Normal file
View file

@ -0,0 +1,116 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\CliMulti;
use Piwik\Common;
use Piwik\Config;
class CliPhp
{
public function findPhpBinary()
{
$general = Config::getInstance()->General;
if (!empty($general['php_binary_path']) && file_exists($general['php_binary_path'])) {
return $general['php_binary_path'];
}
if (defined('PHP_BINARY')) {
if ($this->isHhvmBinary(PHP_BINARY)) {
return PHP_BINARY . ' --php';
}
if ($this->isValidPhpType(PHP_BINARY)) {
return PHP_BINARY . ' -q';
}
}
$bin = '';
if (!empty($_SERVER['_']) && Common::isPhpCliMode()) {
$bin = $this->getPhpCommandIfValid($_SERVER['_']);
}
if (empty($bin) && !empty($_SERVER['argv'][0]) && Common::isPhpCliMode()) {
$bin = $this->getPhpCommandIfValid($_SERVER['argv'][0]);
}
if (empty($bin)) {
$possiblePhpPath = PHP_BINDIR . ('\\' === \DIRECTORY_SEPARATOR ? '\\php.exe' : '/php');
$bin = $this->getPhpCommandIfValid($possiblePhpPath);
}
if (!$this->isValidPhpType($bin) && function_exists('shell_exec')) {
$bin = @shell_exec('which php');
}
if (!$this->isValidPhpType($bin)) {
return false;
}
$bin = trim($bin);
if (!$this->isValidPhpVersion($bin)) {
return false;
}
$bin .= ' -q';
return $bin;
}
private function isHhvmBinary($bin)
{
return false !== strpos($bin, 'hhvm');
}
private function isValidPhpVersion($bin)
{
global $piwik_minimumPHPVersion;
$cliVersion = $this->getPhpVersion($bin);
$isCliVersionValid = $cliVersion && version_compare($piwik_minimumPHPVersion, $cliVersion) <= 0;
return $isCliVersionValid;
}
private function isValidPhpType($path)
{
if (empty($path)) {
return false;
}
$path = basename($path);
return false === strpos($path, 'fpm')
&& false === strpos($path, 'cgi')
&& false === strpos($path, 'phpunit')
&& false === strpos($path, 'lsphp');
}
private function getPhpCommandIfValid($path)
{
if (!empty($path) && @is_executable($path)) {
if (0 === strpos($path, PHP_BINDIR) && $this->isValidPhpType($path)) {
return $path;
}
}
return null;
}
/**
* @param string $bin PHP binary
* @return string
*/
private function getPhpVersion($bin)
{
$command = sprintf("%s -r 'echo phpversion();'", $bin);
$version = null;
if (function_exists('shell_exec')) {
$version = @shell_exec($command);
}
return $version;
}
}

75
matomo/core/CliMulti/Output.php Normal file
View file

@ -0,0 +1,75 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\CliMulti;
use Piwik\CliMulti;
use Piwik\Filesystem;
class Output implements OutputInterface
{
private $tmpFile = '';
private $outputId = null;
public function __construct($outputId)
{
if (!Filesystem::isValidFilename($outputId)) {
throw new \Exception('The given output id has an invalid format');
}
$dir = CliMulti::getTmpPath();
Filesystem::mkdir($dir);
$this->tmpFile = $dir . '/' . $outputId . '.output';
$this->outputId = $outputId;
}
public function getOutputId()
{
return $this->outputId;
}
public function write($content)
{
file_put_contents($this->tmpFile, $content);
}
public function getPathToFile()
{
return $this->tmpFile;
}
public function isAbnormal(): bool
{
$size = Filesystem::getFileSize($this->tmpFile, 'MB');
return $size !== null && $size >= 100;
}
public function exists(): bool
{
return file_exists($this->tmpFile);
}
public function get()
{
$content = @file_get_contents($this->tmpFile);
$search = '#!/usr/bin/env php';
if (!empty($content)
&& is_string($content)
&& mb_substr(trim($content), 0, strlen($search)) === $search) {
$content = trim(mb_substr(trim($content), strlen($search)));
}
return $content;
}
public function destroy()
{
Filesystem::deleteFileIfExists($this->tmpFile);
}
}

24
matomo/core/CliMulti/OutputInterface.php Normal file
View file

@ -0,0 +1,24 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\CliMulti;
interface OutputInterface
{
public function getOutputId();
public function write($content);
public function isAbnormal(): bool;
public function exists(): bool;
public function get();
public function destroy();
}

323
matomo/core/CliMulti/Process.php Normal file
View file

@ -0,0 +1,323 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\CliMulti;
use Piwik\CliMulti;
use Piwik\Common;
use Piwik\Container\StaticContainer;
use Piwik\Filesystem;
use Piwik\SettingsServer;
/**
* There are three different states
* - PID file exists with empty content: Process is created but not started
* - PID file exists with the actual process PID as content: Process is running
* - PID file does not exist: Process is marked as finished
*
* Class Process
*/
class Process
{
const PS_COMMAND = 'ps x';
const AWK_COMMAND = 'awk \'! /defunct/ {print $1}\'';
private $finished = null;
private $pidFile = '';
private $timeCreation = null;
private static $isSupported = null;
private $pid = null;
private $started = null;
public function __construct($pid)
{
if (!Filesystem::isValidFilename($pid)) {
throw new \Exception('The given pid has an invalid format');
}
$pidDir = CliMulti::getTmpPath();
Filesystem::mkdir($pidDir);
$this->pidFile = $pidDir . '/' . $pid . '.pid';
$this->timeCreation = time();
$this->pid = $pid;
$this->markAsNotStarted();
}
private static function isForcingAsyncProcessMode()
{
try {
return (bool) StaticContainer::get('test.vars.forceCliMultiViaCurl');
} catch (\Exception $ex) {
return false;
}
}
public function getPid()
{
return $this->pid;
}
private function markAsNotStarted()
{
$content = $this->getPidFileContent();
if ($this->doesPidFileExist($content)) {
return;
}
$this->writePidFileContent('');
}
public function hasStarted($content = null)
{
if (!$this->started) {
$this->started = $this->checkPidIfHasStarted($content);
}
// PID will be deleted when process has finished so we want to remember this process started at some point. Otherwise we might return false here once the process finished.
// therefore we want to "cache" a successful start
return $this->started;
}
private function checkPidIfHasStarted($content = null)
{
if (is_null($content)) {
$content = $this->getPidFileContent();
}
if (!$this->doesPidFileExist($content)) {
// process is finished, this means there was a start before
return true;
}
if ('' === trim($content)) {
// pid file is overwritten by startProcess()
return false;
}
// process is probably running or pid file was not removed
return true;
}
public function hasFinished()
{
if ($this->finished) {
return true;
}
$content = $this->getPidFileContent();
return !$this->doesPidFileExist($content);
}
public function getSecondsSinceCreation()
{
return time() - $this->timeCreation;
}
public function startProcess()
{
$this->writePidFileContent(getmypid());
}
public function isRunning()
{
$content = $this->getPidFileContent();
if (!$this->doesPidFileExist($content)) {
return false;
}
if (!$this->pidFileSizeIsNormal($content)) {
$this->finishProcess();
return false;
}
if ($this->isProcessStillRunning($content)) {
return true;
}
if ($this->hasStarted($content)) {
$this->finishProcess();
}
return false;
}
private function pidFileSizeIsNormal($content)
{
$size = Common::mb_strlen($content);
return $size < 500;
}
public function finishProcess()
{
$this->finished = true;
Filesystem::deleteFileIfExists($this->pidFile);
}
private function doesPidFileExist($content)
{
return false !== $content;
}
private function isProcessStillRunning($content)
{
if (!self::isSupported()) {
return true;
}
$lockedPID = trim($content);
$runningPIDs = self::getRunningProcesses();
return !empty($lockedPID) && in_array($lockedPID, $runningPIDs);
}
private function getPidFileContent()
{
return @file_get_contents($this->pidFile);
}
/**
* Tests only
* @internal
* @param $content
*/
public function writePidFileContent($content)
{
file_put_contents($this->pidFile, $content);
}
public static function isSupported()
{
if (!isset(self::$isSupported)) {
$reasons = self::isSupportedWithReason();
self::$isSupported = empty($reasons);
}
return self::$isSupported;
}
public static function isSupportedWithReason()
{
$reasons = [];
if (defined('PIWIK_TEST_MODE')
&& self::isForcingAsyncProcessMode()
) {
$reasons[] = 'forcing multicurl use for tests';
}
if (SettingsServer::isWindows()) {
$reasons[] = 'not supported on windows';
return $reasons;
}
if (self::isMethodDisabled('shell_exec')) {
$reasons[] = 'shell_exec is disabled';
return $reasons; // shell_exec is used for almost every other check
}
$getMyPidDisabled = self::isMethodDisabled('getmypid');
if ($getMyPidDisabled) {
$reasons[] = 'getmypid is disabled';
}
if (self::isSystemNotSupported()) {
$reasons[] = 'system returned by `uname -a` is not supported';
}
if (!self::psExistsAndRunsCorrectly()) {
$reasons[] = 'shell_exec(' . self::PS_COMMAND . '" 2> /dev/null") did not return a success code';
} else if (!$getMyPidDisabled) {
$pid = @getmypid();
if (empty($pid) || !in_array($pid, self::getRunningProcesses())) {
$reasons[] = 'could not find our pid (from getmypid()) in the output of `' . self::PS_COMMAND . '`';
}
}
if (!self::awkExistsAndRunsCorrectly()) {
$reasons[] = 'awk is not available or did not run as we would expect it to';
}
return $reasons;
}
private static function psExistsAndRunsCorrectly()
{
return self::returnsSuccessCode(self::PS_COMMAND . ' 2>/dev/null');
}
private static function awkExistsAndRunsCorrectly()
{
$testResult = @shell_exec('echo " 537 s000 Ss 0:00.05 login -pfl theuser /bin/bash -c exec -la bash /bin/bash" | ' . self::AWK_COMMAND . ' 2>/dev/null');
return trim($testResult) == '537';
}
private static function isSystemNotSupported()
{
$uname = @shell_exec('uname -a 2> /dev/null');
if (empty($uname)) {
$uname = php_uname();
}
if (strpos($uname, 'synology') !== false) {
return true;
}
return false;
}
public static function isMethodDisabled($command)
{
if (!function_exists($command)) {
return true;
}
$disabled = explode(',', ini_get('disable_functions'));
$disabled = array_map('trim', $disabled);
return in_array($command, $disabled) || !function_exists($command);
}
private static function returnsSuccessCode($command)
{
$exec = $command . ' > /dev/null 2>&1; echo $?';
$returnCode = @shell_exec($exec);
if (false === $returnCode || null === $returnCode) {
return false;
}
$returnCode = trim($returnCode);
return 0 == (int) $returnCode;
}
public static function getListOfRunningProcesses()
{
$processes = @shell_exec(self::PS_COMMAND . ' 2>/dev/null');
if (empty($processes)) {
return array();
}
return explode("\n", $processes);
}
/**
* @return int[] The ids of the currently running processes
*/
public static function getRunningProcesses()
{
$ids = explode("\n", trim(shell_exec(self::PS_COMMAND . ' 2>/dev/null | ' . self::AWK_COMMAND . ' 2>/dev/null')));
$ids = array_map('intval', $ids);
$ids = array_filter($ids, function ($id) {
return $id > 0;
});
return $ids;
}
}

134
matomo/core/CliMulti/RequestCommand.php Normal file
View file

@ -0,0 +1,134 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\CliMulti;
use Piwik\Application\Environment;
use Piwik\Access;
use Piwik\Container\StaticContainer;
use Piwik\Db;
use Piwik\Log;
use Piwik\Option;
use Piwik\Plugin\ConsoleCommand;
use Piwik\Url;
use Piwik\UrlHelper;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
/**
* RequestCommand
*/
class RequestCommand extends ConsoleCommand
{
/**
* @var Environment
*/
private $environment;
protected function configure()
{
$this->setName('climulti:request');
$this->setDescription('Parses and executes the given query. See Piwik\CliMulti. Intended only for system usage.');
$this->addArgument('url-query', InputArgument::REQUIRED, 'Matomo URL query string, for instance: "module=API&method=API.getPiwikVersion&token_auth=123456789"');
$this->addOption('superuser', null, InputOption::VALUE_NONE, 'If supplied, runs the code as superuser.');
}
protected function execute(InputInterface $input, OutputInterface $output)
{
$this->recreateContainerWithWebEnvironment();
$this->initHostAndQueryString($input);
if ($this->isTestModeEnabled()) {
$indexFile = '/tests/PHPUnit/proxy/';
$this->resetDatabase();
} else {
$indexFile = '/';
}
$indexFile .= 'index.php';
if (!empty($_GET['pid'])) {
$process = new Process($_GET['pid']);
if ($process->hasFinished()) {
return;
}
$process->startProcess();
}
if ($input->getOption('superuser')) {
StaticContainer::addDefinitions(array(
'observers.global' => \DI\add(array(
array('Environment.bootstrapped', \DI\value(function () {
Access::getInstance()->setSuperUserAccess(true);
}))
)),
));
}
require_once PIWIK_INCLUDE_PATH . $indexFile;
while (ob_get_level()) {
echo ob_get_clean();
}
if (!empty($process)) {
$process->finishProcess();
}
}
private function isTestModeEnabled()
{
return !empty($_GET['testmode']);
}
/**
* @param InputInterface $input
*/
protected function initHostAndQueryString(InputInterface $input)
{
$_GET = array();
$hostname = $input->getOption('matomo-domain');
Url::setHost($hostname);
$query = $input->getArgument('url-query');
$_SERVER['QUERY_STRING'] = $query;
$query = UrlHelper::getArrayFromQueryString($query); // NOTE: this method can create the StaticContainer now
foreach ($query as $name => $value) {
$_GET[$name] = urldecode($value);
}
}
/**
* We will be simulating an HTTP request here (by including index.php).
*
* To avoid weird side-effects (e.g. the logging output messing up the HTTP response on the CLI output)
* we need to recreate the container with the default environment instead of the CLI environment.
*/
private function recreateContainerWithWebEnvironment()
{
StaticContainer::clearContainer();
Log::unsetInstance();
$this->environment = new Environment(null);
$this->environment->init();
}
private function resetDatabase()
{
Option::clearCache();
Db::destroyDatabaseObject();
}
}

112
matomo/core/CliMulti/RequestParser.php Normal file
View file

@ -0,0 +1,112 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\CliMulti;
use Piwik\SettingsPiwik;
class RequestParser
{
private $supportsAsync;
public function __construct($supportsAsync)
{
$this->supportsAsync = $supportsAsync;
}
public function getInProgressCommands()
{
$psOutput = $this->getPsOutput();
$climultiRequestCommands = $this->getPsLinesWithCliMulti($psOutput);
$climultiRequestCommands = $this->parseQueries($climultiRequestCommands);
return $climultiRequestCommands;
}
public function getInProgressArchivingCommands()
{
$commands = $this->getInProgressCommands();
$commands = $this->filterNonArchivingJobs($commands);
return $commands;
}
private function getPsOutput() // protected for tests
{
if (!$this->supportsAsync) {
// we cannot detect if web archive is still running
return '';
}
return $this->invokePs();
}
private function filterNonArchivingJobs($commands)
{
$result = array_filter($commands, function ($command) {
if (empty($command['trigger'])
|| $command['trigger'] != 'archivephp'
) {
return false;
}
return true;
});
$result = array_values($result);
return $result;
}
private function getPsLinesWithCliMulti(string $psOutput)
{
$instanceId = SettingsPiwik::getPiwikInstanceId();
$lines = explode("\n", $psOutput);
$lines = array_map('trim', $lines);
$lines = array_filter($lines, function ($line) use ($instanceId) {
if (!empty($instanceId) && strpos($line, 'matomo-domain=' . $instanceId) === false) {
return false;
}
return strpos($line, 'climulti:request') !== false
&& (
strpos($line, 'console') !== false || strpos($line, 'php') !== false
);
});
return $lines;
}
private function parseQueries(array $climultiRequestCommands)
{
$commandName = 'climulti:request';
$result = [];
foreach ($climultiRequestCommands as $command) {
$pos = strpos($command, $commandName);
$commandParts = substr($command, $pos + strlen($commandName));
$commandParts = explode(" ", $commandParts);
$commandParts = array_filter($commandParts, function ($p) {
return strlen($p) && substr($p, 0, 1) != '-';
});
$query = reset($commandParts);
parse_str($query, $parsed);
$result[] = $parsed;
}
return $result;
}
protected function invokePs()
{
if (defined('PIWIK_TEST_MODE')) {
return ''; // skip check in tests as it might result in random failures
}
return `ps aux`;
}
}

59
matomo/core/CliMulti/StaticOutput.php Normal file
View file

@ -0,0 +1,59 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*/
namespace Piwik\CliMulti;
use Piwik\Common;
class StaticOutput implements OutputInterface
{
private $content = false;
private $outputId = null;
public function __construct($outputId)
{
$this->outputId = $outputId;
}
public function getOutputId()
{
return $this->outputId;
}
public function write($content)
{
$this->content = (string) $content;
}
public function getPathToFile()
{
return '';
}
public function isAbnormal(): bool
{
$size = Common::mb_strlen($this->content);
$hundredMb = 100 * 1024 * 1024;
return $size >= $hundredMb;
}
public function exists(): bool
{
return $this->content !== false;
}
public function get()
{
return $this->content;
}
public function destroy()
{
$this->content = false;
}
}

55
matomo/core/Columns/ComputedMetricFactory.php Normal file
View file

@ -0,0 +1,55 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Columns;
use Piwik\Plugin\ArchivedMetric;
use Piwik\Plugin\ComputedMetric;
/**
* A factory to create computed metrics.
*
* @api since Piwik 3.2.0
*/
class ComputedMetricFactory
{
/**
* @var MetricsList
*/
private $metricsList = null;
/**
* Generates a new report metric factory.
* @param MetricsList $list A report list instance
* @ignore
*/
public function __construct(MetricsList $list)
{
$this->metricsList = $list;
}
/**
* @return \Piwik\Plugin\ComputedMetric
*/
public function createComputedMetric($metricName1, $metricName2, $aggregation)
{
$metric1 = $this->metricsList->getMetric($metricName1);
if (!$metric1 instanceof ArchivedMetric || !$metric1->getDimension()) {
throw new \Exception('Only possible to create computed metric for an archived metric with a dimension');
}
$dimension1 = $metric1->getDimension();
$metric = new ComputedMetric($metricName1, $metricName2, $aggregation);
$metric->setCategory($dimension1->getCategoryId());
return $metric;
}
}

856
matomo/core/Columns/Dimension.php Normal file
View file

@ -0,0 +1,856 @@
<?php
/**
* Matomo - free/libre analytics platform
*
* @link https://matomo.org
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
*
*/
namespace Piwik\Columns;
use Piwik\Common;
use Piwik\Db;
use Piwik\Piwik;
use Piwik\Plugin;
use Piwik\Plugin\ArchivedMetric;
use Piwik\Plugin\Segment;
use Exception;
use Piwik\CacheId;
use Piwik\Cache as PiwikCache;
use Piwik\Plugin\Manager as PluginManager;
use Piwik\Metrics\Formatter;
use Piwik\Segment\SegmentsList;
/**
* @api
* @since 3.1.0
*/
abstract class Dimension
{
const COMPONENT_SUBNAMESPACE = 'Columns';
/**
* Segment type 'dimension'. Can be used along with {@link setType()}.
* @api
*/
const TYPE_DIMENSION = 'dimension';
const TYPE_BINARY = 'binary';
const TYPE_TEXT = 'text';
const TYPE_ENUM = 'enum';
const TYPE_MONEY = 'money';
const TYPE_BYTE = 'byte';
const TYPE_DURATION_MS = 'duration_ms';
const TYPE_DURATION_S = 'duration_s';
const TYPE_NUMBER = 'number';
const TYPE_FLOAT = 'float';
const TYPE_URL = 'url';
const TYPE_DATE = 'date';
const TYPE_TIME = 'time';
const TYPE_DATETIME = 'datetime';
const TYPE_TIMESTAMP = 'timestamp';
const TYPE_BOOL = 'bool';
const TYPE_PERCENT = 'percent';
/**
* This will be the name of the column in the database table if a $columnType is specified.
* @var string
* @api
*/
protected $columnName = '';
/**
* If a columnType is defined, we will create a column in the MySQL table having this type. Please make sure
* MySQL understands this type. Once you change the column type the Piwik platform will notify the user to
* perform an update which can sometimes take a long time so be careful when choosing the correct column type.
* @var string
* @api
*/
protected $columnType = '';
/**
* Holds an array of segment instances
* @var Segment[]
*/
protected $segments = array();
/**
* Defines what kind of data type this dimension holds. By default the type is auto-detected based on
* `$columnType` but sometimes it may be needed to correct this value. Depending on this type, a dimension will be
* formatted differently for example.
* @var string
* @api since Piwik 3.2.0
*/
protected $type = '';
/**
* Translation key for name singular
* @var string
*/
protected $nameSingular = '';
/**
* Translation key for name plural
* @var string
* @api since Piwik 3.2.0
*/
protected $namePlural = '';
/**
* Translation key for category
* @var string
*/
protected $category = '';
/**
* By defining a segment name a user will be able to filter their visitors by this column. If you do not want to
* define a segment for this dimension, simply leave the name empty.
* @api since Piwik 3.2.0
*/
protected $segmentName = '';
/**
* Sets a callback which will be executed when user will call for suggested values for segment.
*
* @var callable
* @api since Piwik 3.2.0
*/
protected $suggestedValuesCallback;
/**
* An API method whose label columns will be used to determine the suggested values should browser archiving
* be disabled. The API must have defined a segment metadata on each row for this to work.
* @var string
*/
protected $suggestedValuesApi = '';
/**
* Here you should explain which values are accepted/useful for your segment, for example:
* "1, 2, 3, etc." or "comcast.net, proxad.net, etc.". If the value needs any special encoding you should mention
* this as well. For example "Any URL including protocol. The URL must be URL encoded."
*
* @var string
* @api since Piwik 3.2.0
*/
protected $acceptValues;
/**
* Defines to which column in the MySQL database the segment belongs (if one is configured). Defaults to
* `$this.dbTableName . '.'. $this.columnName` but you can customize it eg like `HOUR(log_visit.visit_last_action_time)`.
*
* @param string $sqlSegment
* @api since Piwik 3.2.0
*/
protected $sqlSegment;
/**
* Interesting when specifying a segment. Sometimes you want users to set segment values that differ from the way
* they are actually stored. For instance if you want to allow to filter by any URL than you might have to resolve
* this URL to an action id. Or a country name maybe has to be mapped to a 2 letter country code. You can do this by
* specifying either a callable such as `array('Classname', 'methodName')` or by passing a closure.
* There will be four values passed to the given closure or callable: `string $valueToMatch`, `string $segment`
* (see {@link setSegment()}), `string $matchType` (eg SegmentExpression::MATCH_EQUAL or any other match constant
* of this class) and `$segmentName`.
*
* If the closure returns NULL, then Piwik assumes the segment sub-string will not match any visitor.
*
* @var string|\Closure
* @api since Piwik 3.2.0
*/
protected $sqlFilter;
/**
* Similar to {@link $sqlFilter} you can map a given segment value to another value. For instance you could map
* "new" to 0, 'returning' to 1 and any other value to '2'. You can either define a callable or a closure. There
* will be only one value passed to the closure or callable which contains the value a user has set for this
* segment.
* @var string|array
* @api since Piwik 3.2.0
*/
protected $sqlFilterValue;
/**
* Defines whether this dimension (and segment based on this dimension) is available to anonymous users.
* @var bool
* @api since Piwik 3.2.0
*/
protected $allowAnonymous = true;
/**
* The name of the database table this dimension refers to
* @var string
* @api
*/
protected $dbTableName = '';
/**
* By default the metricId is automatically generated based on the dimensionId. This might sometimes not be as
* readable and quite long. If you want more expressive metric names like `nb_visits` compared to
* `nb_corehomevisitid`, you can eg set a metricId `visit`.
*
* @var string
* @api since Piwik 3.2.0
*/
protected $metricId = '';
/**
* To be implemented when a column references another column
* @return Join|null
* @api since Piwik 3.2.0
*/
public function getDbColumnJoin()
{
return null;
}
/**
* @return Discriminator|null
* @api since Piwik 3.2.0
*/
public function getDbDiscriminator()
{
return null;
}
/**
* To be implemented when a column represents an enum.
* @return array
* @api since Piwik 3.2.0
*/
public function getEnumColumnValues()
{
return array();
}
/**
* Get the metricId which is used to generate metric names based on this dimension.
* @return string
*/
public function getMetricId()
{
if (!empty($this->metricId)) {
return $this->metricId;
}
$id = $this->getId();
return str_replace(array('.', ' ', '-'), '_', strtolower($id));
}
/**
* Installs the action dimension in case it is not installed yet. The installation is already implemented based on
* the {@link $columnName} and {@link $columnType}. If you want to perform additional actions beside adding the
* column to the database - for instance adding an index - you can overwrite this method. We recommend to call
* this parent method to get the minimum required actions and then add further custom actions since this makes sure
* the column will be installed correctly. We also recommend to change the default install behavior only if really
* needed. FYI: We do not directly execute those alter table statements here as we group them together with several
* other alter table statements do execute those changes in one step which results in a faster installation. The
* column will be added to the `log_link_visit_action` MySQL table.
*
* Example:
* ```
public function install()
{
$changes = parent::install();
$changes['log_link_visit_action'][] = "ADD INDEX index_idsite_servertime ( idsite, server_time )";
return $changes;
}
```
*
* @return array An array containing the table name as key and an array of MySQL alter table statements that should
* be executed on the given table. Example:
* ```
array(
'log_link_visit_action' => array("ADD COLUMN `$this->columnName` $this->columnType", "ADD INDEX ...")
);
```
* @api
*/
public function install()
{
if (empty($this->columnName) || empty($this->columnType) || empty($this->dbTableName)) {
return array();
}
// TODO if table does not exist, create it with a primary key, but at this point we cannot really create it
// cause we need to show the query in the UI first and user needs to be able to create table manually.
// we cannot return something like "create table " here as it would be returned for each table etc.
// we need to do this in column updater etc!
return array(
$this->dbTableName => array("ADD COLUMN `$this->columnName` $this->columnType")
);
}
/**
* Updates the action dimension in case the {@link $columnType} has changed. The update is already implemented based
* on the {@link $columnName} and {@link $columnType}. This method is intended not to overwritten by plugin
* developers as it is only supposed to make sure the column has the correct type. Adding additional custom "alter
* table" actions would not really work since they would be executed with every {@link $columnType} change. So
* adding an index here would be executed whenever the columnType changes resulting in an error if the index already
* exists. If an index needs to be added after the first version is released a plugin update class should be
* created since this makes sure it is only executed once.
*
* @return array An array containing the table name as key and an array of MySQL alter table statements that should
* be executed on the given table. Example:
* ```
array(
'log_link_visit_action' => array("MODIFY COLUMN `$this->columnName` $this->columnType", "DROP COLUMN ...")
);
```
* @ignore
*/
public function update()
{
if (empty($this->columnName) || empty($this->columnType) || empty($this->dbTableName)) {
return array();
}
return array(
$this->dbTableName => array("MODIFY COLUMN `$this->columnName` $this->columnType")
);
}
/**
* Uninstalls the dimension if a {@link $columnName} and {@link columnType} is set. In case you perform any custom
* actions during {@link install()} - for instance adding an index - you should make sure to undo those actions by
* overwriting this method. Make sure to call this parent method to make sure the uninstallation of the column
* will be done.
* @throws Exception
* @api
*/
public function uninstall()
{
if (empty($this->columnName) || empty($this->columnType) || empty($this->dbTableName)) {
return;
}
try {
$sql = "ALTER TABLE `" . Common::prefixTable($this->dbTableName) . "` DROP COLUMN `$this->columnName`";
Db::exec($sql);
} catch (Exception $e) {
if (!Db::get()->isErrNo($e, '1091')) {
throw $e;
}
}
}
/**
* Returns the ID of the category (typically a translation key).
* @return string
*/
public function getCategoryId()
{
return $this->category;
}
/**
* Returns the translated name of this dimension which is typically in singular.
*
* @return string
*/
public function getName()
{
if (!empty($this->nameSingular)) {
return Piwik::translate($this->nameSingular);
}
return $this->nameSingular;
}
/**
* Returns a translated name in plural for this dimension.
* @return string
* @api since Piwik 3.2.0
*/
public function getNamePlural()
{
if (!empty($this->namePlural)) {
return Piwik::translate($this->namePlural);
}
return $this->getName();
}
/**
* Defines whether an anonymous user is allowed to view this dimension
* @return bool
* @api since Piwik 3.2.0
*/
public function isAnonymousAllowed()
{
return $this->allowAnonymous;
}
/**
* Sets (overwrites) the SQL segment
* @param $segment
* @api since Piwik 3.2.0
*/
public function setSqlSegment($segment)
{
$this->sqlSegment = $segment;
}
/**
* Sets (overwrites the dimension type)
* @param $type
* @api since Piwik 3.2.0
*/
public function setType($type)
{
$this->type = $type;
}
/**
* A dimension should group values by using this method. Otherwise the same row may appear several times.
*
* @param mixed $value
* @param int $idSite
* @return mixed
* @api since Piwik 3.2.0
*/
public function groupValue($value, $idSite)
{
switch ($this->type) {
case Dimension::TYPE_URL:
return str_replace(array('http://', 'https://'), '', $value);
case Dimension::TYPE_BOOL:
return !empty($value) ? '1' : '0';
case Dimension::TYPE_DURATION_MS:
return number_format($value / 1000, 2) * 1000; // because we divide we need to group them and cannot do this in formatting step
}
return $value;
}
/**
* Formats the dimension value. By default, the dimension is formatted based on the set dimension type.
*
* @param mixed $value
* @param int $idSite
* @param Formatter $formatter
* @return mixed
* @api since Piwik 3.2.0
*/
public function formatValue($value, $idSite, Formatter $formatter)
{
switch ($this->type) {
case Dimension::TYPE_BOOL:
if (empty($value)) {
return Piwik::translate('General_No');
}
return Piwik::translate('General_Yes');
case Dimension::TYPE_ENUM:
$values = $this->getEnumColumnValues();
if (isset($values[$value])) {
return $values[$value];
}
break;
case Dimension::TYPE_MONEY:
return $formatter->getPrettyMoney($value, $idSite);
case Dimension::TYPE_FLOAT:
return $formatter->getPrettyNumber((float) $value, $precision = 2);
case Dimension::TYPE_NUMBER:
return $formatter->getPrettyNumber($value);
case Dimension::TYPE_DURATION_S:
return $formatter->getPrettyTimeFromSeconds($value, $displayAsSentence = false);
case Dimension::TYPE_DURATION_MS:
$val = round(($value / 1000), ($value / 1000) > 60 ? 0 : 2);
return $formatter->getPrettyTimeFromSeconds($val, $displayAsSentence = true);
case Dimension::TYPE_PERCENT:
return $formatter->getPrettyPercentFromQuotient($value);
case Dimension::TYPE_BYTE:
return $formatter->getPrettySizeFromBytes($value);
}
return $value;
}
/**
* Overwrite this method to configure segments. To do so just create an instance of a {@link \Piwik\Plugin\Segment}
* class, configure it and call the {@link addSegment()} method. You can add one or more segments for this
* dimension. Example:
*
* ```
* $segment = new Segment();
* $segment->setSegment('exitPageUrl');
* $segment->setName('Actions_ColumnExitPageURL');
* $segment->setCategory('General_Visit');
* $segmentsList->addSegment($segment);
* ```
*
* @param SegmentsList $segmentsList
* @param DimensionSegmentFactory $dimensionSegmentFactory
* @throws Exception
*/
public function configureSegments(SegmentsList $segmentsList, DimensionSegmentFactory $dimensionSegmentFactory)
{
if ($this->segmentName && $this->category
&& ($this->sqlSegment || ($this->columnName && $this->dbTableName))
&& $this->nameSingular) {
$segment = $dimensionSegmentFactory->createSegment(null);
$segmentsList->addSegment($segment);
}
}
/**
* Configures metrics for this dimension.
*
* For certain dimension types, some metrics will be added automatically.
*
* @param MetricsList $metricsList
* @param DimensionMetricFactory $dimensionMetricFactory
*/
public function configureMetrics(MetricsList $metricsList, DimensionMetricFactory $dimensionMetricFactory)
{
if ($this->getMetricId() && $this->dbTableName && $this->columnName && $this->getNamePlural()) {
if (in_array($this->getType(), array(self::TYPE_DATETIME, self::TYPE_DATE, self::TYPE_TIME, self::TYPE_TIMESTAMP))) {
// we do not generate any metrics from these types
return;
} elseif (in_array($this->getType(), array(self::TYPE_URL, self::TYPE_TEXT, self::TYPE_BINARY, self::TYPE_ENUM))) {
$metric = $dimensionMetricFactory->createMetric(ArchivedMetric::AGGREGATION_UNIQUE);
$metricsList->addMetric($metric);
} elseif (in_array($this->getType(), array(self::TYPE_BOOL))) {
$metric = $dimensionMetricFactory->createMetric(ArchivedMetric::AGGREGATION_SUM);
$metricsList->addMetric($metric);
} else {
$metric = $dimensionMetricFactory->createMetric(ArchivedMetric::AGGREGATION_SUM);
$metricsList->addMetric($metric);
$metric = $dimensionMetricFactory->createMetric(ArchivedMetric::AGGREGATION_MAX);
$metricsList->addMetric($metric);
}
}
}
/**
* Check whether a dimension has overwritten a specific method.
* @param $method
* @return bool
* @ignore
*/
public function hasImplementedEvent($method)
{
$method = new \ReflectionMethod($this, $method);
$declaringClass = $method->getDeclaringClass();
return 0 === strpos($declaringClass->name, 'Piwik\Plugins');
}
/**
* Get the list of configured segments.
*
* @return Segment[]
* @throws Exception
* @ignore
*/
public function getSegments()
{
$list = new SegmentsList();
$this->configureSegments($list, new DimensionSegmentFactory($this));
return $list->getSegments();
}
/**
* Returns the name of the segment that this dimension defines
* @return string
* @api since Piwik 3.2.0
*/
public function getSegmentName()
{
return $this->segmentName;
}
/**
* Get the name of the dimension column.
* @return string
* @ignore
*/
public function getColumnName()
{
return $this->columnName;
}
/**
* Returns a sql segment expression for this dimension.
* @return string
* @api since Piwik 3.2.0
*/
public function getSqlSegment()
{
if (!empty($this->sqlSegment)) {
return $this->sqlSegment;
}
if ($this->dbTableName && $this->columnName) {
return $this->dbTableName . '.' . $this->columnName;
}
}
/**
* @return null|callable
* @ignore
*/
public function getSuggestedValuesCallback()
{
return $this->suggestedValuesCallback;
}
/**
* @return null|string
* @ignore
*/
public function getSuggestedValuesApi()
{
return $this->suggestedValuesApi;
}
/**
* @return null|string
* @ignore
*/
public function getAcceptValues()
{
return $this->acceptValues;
}
/**
* @return \Closure|string|null
* @ignore
*/
public function getSqlFilter()
{
return $this->sqlFilter;
}
/**
* @return array|string|null
* @ignore
*/
public function getSqlFilterValue()
{
return $this->sqlFilterValue;
}
/**
* Check whether the dimension has a column type configured
* @return bool
* @ignore
*/
public function hasColumnType()
{
return !empty($this->columnType);
}
/**
* Returns the name of the database table this dimension belongs to.
* @return string
* @api since Piwik 3.2.0
*/
public function getDbTableName()
{
return $this->dbTableName;
}
/**
* Returns a unique string ID for this dimension. The ID is built using the namespaced class name
* of the dimension, but is modified to be more human readable.
*
* @return string eg, `"Referrers.Keywords"`
* @throws Exception if the plugin and simple class name of this instance cannot be determined.
* This would only happen if the dimension is located in the wrong directory.
* @api
*/
public function getId()
{
$className = get_class($this);
return $this->generateIdFromClass($className);
}
/**
* @param string $className
* @return string
* @throws Exception
* @ignore
*/
protected function generateIdFromClass($className)
{
// parse plugin name & dimension name
$regex = "/Piwik\\\\Plugins\\\\([^\\\\]+)\\\\" . self::COMPONENT_SUBNAMESPACE . "\\\\([^\\\\]+)/";
if (!preg_match($regex, $className, $matches)) {
throw new Exception("'$className' is located in the wrong directory.");
}
$pluginName = $matches[1];
$dimensionName = $matches[2];
return $pluginName . '.' . $dimensionName;
}
/**
* Gets an instance of all available visit, action and conversion dimension.
* @return Dimension[]
*/
public static function getAllDimensions()
{
$cacheId = CacheId::siteAware(CacheId::pluginAware('AllDimensions'));
$cache = PiwikCache::getTransientCache();
if (!$cache->contains($cacheId)) {
$plugins = PluginManager::getInstance()->getPluginsLoadedAndActivated();
$instances = array();
/**
* Triggered to add new dimensions that cannot be picked up automatically by the platform.
* This is useful if the plugin allows a user to create reports / dimensions dynamically. For example
* CustomDimensions or CustomVariables. There are a variable number of dimensions in this case and it
* wouldn't be really possible to create a report file for one of these dimensions as it is not known
* how many Custom Dimensions will exist.
*
* **Example**
*
* public function addDimension(&$dimensions)
* {
* $dimensions[] = new MyCustomDimension();
* }
*
* @param Dimension[] $reports An array of dimensions
*/
Piwik::postEvent('Dimension.addDimensions', array(&$instances));
foreach ($plugins as $plugin) {
foreach (self::getDimensions($plugin) as $instance) {
$instances[] = $instance;
}
}
/**
* Triggered to filter / restrict dimensions.
*
* **Example**
*
* public function filterDimensions(&$dimensions)
* {
* foreach ($dimensions as $index => $dimension) {
* if ($dimension->getName() === 'Page URL') {}
* unset($dimensions[$index]); // remove this dimension
* }
* }
* }
*
* @param Dimension[] $dimensions An array of dimensions
*/
Piwik::postEvent('Dimension.filterDimensions', array(&$instances));
$cache->save($cacheId, $instances);
}
return $cache->fetch($cacheId);
}
public static function getDimensions(Plugin $plugin)
{
$columns = $plugin->findMultipleComponents('Columns', '\\Piwik\\Columns\\Dimension');
$instances = array();
$removedDimensions = self::getRemovedDimensions();
foreach ($columns as $column) {
if (!in_array($column, $removedDimensions)) {
$instances[] = new $column();
}
}
return $instances;
}
/**
* Returns a list of dimension class names that have been removed from core over time
*
* @return string[]
*/
public static function getRemovedDimensions()
{
return [
// dimensions removed in Matomo 4.0.0
'Piwik\Plugins\DevicePlugins\Columns\PluginDirector',
'Piwik\Plugins\DevicePlugins\Columns\PluginGears',
'Piwik\Plugins\VisitorInterest\Columns\VisitsByDaysSinceLastVisit',
];
}
/**
* Returns the name of the plugin that contains this Dimension.
*
* @return string
* @throws Exception if the Dimension is not located within a Plugin module.
* @api
*/
public function getModule()
{
$id = $this->getId();
if (empty($id)) {
throw new Exception("Invalid dimension ID: '$id'.");
}
$parts = explode('.', $id);
return reset($parts);
}
/**
* Returns the type of the dimension which defines what kind of value this dimension stores.
* @return string
* @api since Piwik 3.2.0
*/
public function getType()
{
if (!empty($this->type)) {
return $this->type;
}
if ($this->getDbColumnJoin()) {
// best guess
return self::TYPE_TEXT;
}
if ($this->getEnumColumnValues()) {
// best guess
return self::TYPE_ENUM;
}
if (!empty($this->columnType)) {
// best guess
$type = strtolower($this->columnType);
if (strpos($type, 'datetime') !== false) {
return self::TYPE_DATETIME;
} elseif (strpos($type, 'timestamp') !== false) {
return self::TYPE_TIMESTAMP;
} elseif (strpos($type, 'date') !== false) {
return self::TYPE_DATE;
} elseif (strpos($type, 'time') !== false) {
return self::TYPE_TIME;
} elseif (strpos($type, 'float') !== false) {
return self::TYPE_FLOAT;
} elseif (strpos($type, 'decimal') !== false) {
return self::TYPE_FLOAT;
} elseif (strpos($type, 'int') !== false) {
return self::TYPE_NUMBER;
} elseif (strpos($type, 'binary') !== false) {
return self::TYPE_BINARY;
}
}
return self::TYPE_TEXT;
}
/**
* Get the version of the dimension which is used for update checks.
* @return string
* @ignore
*/
public function getVersion()
{
return $this->columnType;
}
}

Some files were not shown because too many files have changed in this diff Show more